2023-04-16 00:53:33,504 DEBUG [main] hbase.HBaseTestingUtility(342): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/eff38a88-f310-5857-a834-b48ec32a7488 2023-04-16 00:53:33,517 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.TestAcidGuaranteesWithEagerPolicy timeout: 13 mins 2023-04-16 00:53:33,533 INFO [Time-limited test] hbase.HBaseTestingUtility(1068): Starting up minicluster with option: StartMiniClusterOption{numMasters=1, masterClass=null, numRegionServers=1, rsPorts=, rsClass=null, numDataNodes=1, dataNodeHosts=null, numZkServers=1, createRootDir=false, createWALDir=false} 2023-04-16 00:53:33,533 INFO [Time-limited test] hbase.HBaseZKTestingUtility(82): Created new mini-cluster data directory: /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/eff38a88-f310-5857-a834-b48ec32a7488/cluster_c35e0b5b-fd77-8a80-76dd-b83041c30bf3, deleteOnExit=true 2023-04-16 00:53:33,533 INFO [Time-limited test] hbase.HBaseTestingUtility(1082): STARTING DFS 2023-04-16 00:53:33,534 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting test.cache.data to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/eff38a88-f310-5857-a834-b48ec32a7488/test.cache.data in system properties and HBase conf 2023-04-16 00:53:33,534 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting hadoop.tmp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/eff38a88-f310-5857-a834-b48ec32a7488/hadoop.tmp.dir in system properties and HBase conf 2023-04-16 00:53:33,535 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting hadoop.log.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/eff38a88-f310-5857-a834-b48ec32a7488/hadoop.log.dir in system properties and HBase conf 2023-04-16 00:53:33,535 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.local.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/eff38a88-f310-5857-a834-b48ec32a7488/mapreduce.cluster.local.dir in system properties and HBase conf 2023-04-16 00:53:33,536 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.temp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/eff38a88-f310-5857-a834-b48ec32a7488/mapreduce.cluster.temp.dir in system properties and HBase conf 2023-04-16 00:53:33,536 INFO [Time-limited test] hbase.HBaseTestingUtility(759): read short circuit is OFF 2023-04-16 00:53:33,653 WARN [Time-limited test] util.NativeCodeLoader(62): Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 2023-04-16 00:53:34,068 DEBUG [Time-limited test] fs.HFileSystem(308): The file system is not a DistributedFileSystem. Skipping on block location reordering 2023-04-16 00:53:34,072 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.node-labels.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/eff38a88-f310-5857-a834-b48ec32a7488/yarn.node-labels.fs-store.root-dir in system properties and HBase conf 2023-04-16 00:53:34,073 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.node-attribute.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/eff38a88-f310-5857-a834-b48ec32a7488/yarn.node-attribute.fs-store.root-dir in system properties and HBase conf 2023-04-16 00:53:34,073 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.log-dirs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/eff38a88-f310-5857-a834-b48ec32a7488/yarn.nodemanager.log-dirs in system properties and HBase conf 2023-04-16 00:53:34,073 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/eff38a88-f310-5857-a834-b48ec32a7488/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-04-16 00:53:34,074 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.active-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/eff38a88-f310-5857-a834-b48ec32a7488/yarn.timeline-service.entity-group-fs-store.active-dir in system properties and HBase conf 2023-04-16 00:53:34,074 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.done-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/eff38a88-f310-5857-a834-b48ec32a7488/yarn.timeline-service.entity-group-fs-store.done-dir in system properties and HBase conf 2023-04-16 00:53:34,074 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/eff38a88-f310-5857-a834-b48ec32a7488/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-04-16 00:53:34,074 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/eff38a88-f310-5857-a834-b48ec32a7488/dfs.journalnode.edits.dir in system properties and HBase conf 2023-04-16 00:53:34,075 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.datanode.shared.file.descriptor.paths to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/eff38a88-f310-5857-a834-b48ec32a7488/dfs.datanode.shared.file.descriptor.paths in system properties and HBase conf 2023-04-16 00:53:34,075 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting nfs.dump.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/eff38a88-f310-5857-a834-b48ec32a7488/nfs.dump.dir in system properties and HBase conf 2023-04-16 00:53:34,076 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting java.io.tmpdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/eff38a88-f310-5857-a834-b48ec32a7488/java.io.tmpdir in system properties and HBase conf 2023-04-16 00:53:34,076 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/eff38a88-f310-5857-a834-b48ec32a7488/dfs.journalnode.edits.dir in system properties and HBase conf 2023-04-16 00:53:34,076 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.provided.aliasmap.inmemory.leveldb.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/eff38a88-f310-5857-a834-b48ec32a7488/dfs.provided.aliasmap.inmemory.leveldb.dir in system properties and HBase conf 2023-04-16 00:53:34,077 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting fs.s3a.committer.staging.tmp.path to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/eff38a88-f310-5857-a834-b48ec32a7488/fs.s3a.committer.staging.tmp.path in system properties and HBase conf Formatting using clusterid: testClusterID 2023-04-16 00:53:34,670 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-04-16 00:53:34,674 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-04-16 00:53:34,961 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-namenode.properties,hadoop-metrics2.properties 2023-04-16 00:53:35,145 INFO [Time-limited test] log.Slf4jLog(67): Logging to org.slf4j.impl.Reload4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog 2023-04-16 00:53:35,165 WARN [Time-limited test] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-04-16 00:53:35,203 INFO [Time-limited test] log.Slf4jLog(67): jetty-6.1.26 2023-04-16 00:53:35,234 INFO [Time-limited test] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/hdfs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/eff38a88-f310-5857-a834-b48ec32a7488/java.io.tmpdir/Jetty_localhost_35905_hdfs____.1otfju/webapp 2023-04-16 00:53:35,381 INFO [Time-limited test] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:35905 2023-04-16 00:53:35,419 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-04-16 00:53:35,420 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-04-16 00:53:35,842 WARN [Listener at localhost/43665] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-04-16 00:53:36,000 WARN [Listener at localhost/43665] conf.Configuration(1701): No unit for dfs.datanode.outliers.report.interval(1800000) assuming MILLISECONDS 2023-04-16 00:53:36,022 WARN [Listener at localhost/43665] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-04-16 00:53:36,030 INFO [Listener at localhost/43665] log.Slf4jLog(67): jetty-6.1.26 2023-04-16 00:53:36,036 INFO [Listener at localhost/43665] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/datanode to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/eff38a88-f310-5857-a834-b48ec32a7488/java.io.tmpdir/Jetty_localhost_43917_datanode____rt5i5j/webapp 2023-04-16 00:53:36,137 INFO [Listener at localhost/43665] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:43917 2023-04-16 00:53:36,477 WARN [Listener at localhost/33477] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-04-16 00:53:36,995 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x34e3557c9407f828: Processing first storage report for DS-b52c8e1c-4578-4386-8b7b-56528cefda69 from datanode 1215da9f-09ed-4713-ad5c-9bdf4f6f2014 2023-04-16 00:53:36,997 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x34e3557c9407f828: from storage DS-b52c8e1c-4578-4386-8b7b-56528cefda69 node DatanodeRegistration(127.0.0.1:45865, datanodeUuid=1215da9f-09ed-4713-ad5c-9bdf4f6f2014, infoPort=37039, infoSecurePort=0, ipcPort=33477, storageInfo=lv=-57;cid=testClusterID;nsid=1953208390;c=1681606414745), blocks: 0, hasStaleStorage: true, processing time: 2 msecs, invalidatedBlocks: 0 2023-04-16 00:53:36,998 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x34e3557c9407f828: Processing first storage report for DS-8b1b920e-f1a3-4468-acb4-301b2b0f34f2 from datanode 1215da9f-09ed-4713-ad5c-9bdf4f6f2014 2023-04-16 00:53:36,998 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x34e3557c9407f828: from storage DS-8b1b920e-f1a3-4468-acb4-301b2b0f34f2 node DatanodeRegistration(127.0.0.1:45865, datanodeUuid=1215da9f-09ed-4713-ad5c-9bdf4f6f2014, infoPort=37039, infoSecurePort=0, ipcPort=33477, storageInfo=lv=-57;cid=testClusterID;nsid=1953208390;c=1681606414745), blocks: 0, hasStaleStorage: false, processing time: 0 msecs, invalidatedBlocks: 0 2023-04-16 00:53:37,070 DEBUG [Listener at localhost/33477] hbase.HBaseTestingUtility(649): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/eff38a88-f310-5857-a834-b48ec32a7488 2023-04-16 00:53:37,155 INFO [Listener at localhost/33477] zookeeper.MiniZooKeeperCluster(258): Started connectionTimeout=30000, dir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/eff38a88-f310-5857-a834-b48ec32a7488/cluster_c35e0b5b-fd77-8a80-76dd-b83041c30bf3/zookeeper_0, clientPort=53150, secureClientPort=-1, dataDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/eff38a88-f310-5857-a834-b48ec32a7488/cluster_c35e0b5b-fd77-8a80-76dd-b83041c30bf3/zookeeper_0/version-2, dataDirSize=424 dataLogDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/eff38a88-f310-5857-a834-b48ec32a7488/cluster_c35e0b5b-fd77-8a80-76dd-b83041c30bf3/zookeeper_0/version-2, dataLogSize=424 tickTime=2000, maxClientCnxns=300, minSessionTimeout=4000, maxSessionTimeout=40000, serverId=0 2023-04-16 00:53:37,171 INFO [Listener at localhost/33477] zookeeper.MiniZooKeeperCluster(283): Started MiniZooKeeperCluster and ran 'stat' on client port=53150 2023-04-16 00:53:37,181 INFO [Listener at localhost/33477] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-16 00:53:37,184 INFO [Listener at localhost/33477] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-16 00:53:37,834 INFO [Listener at localhost/33477] util.FSUtils(471): Created version file at hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6 with version=8 2023-04-16 00:53:37,835 INFO [Listener at localhost/33477] hbase.HBaseTestingUtility(1406): Setting hbase.fs.tmp.dir to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/hbase-staging 2023-04-16 00:53:38,130 INFO [Listener at localhost/33477] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2023-04-16 00:53:38,630 INFO [Listener at localhost/33477] client.ConnectionUtils(127): master/jenkins-hbase4:0 server-side Connection retries=45 2023-04-16 00:53:38,661 INFO [Listener at localhost/33477] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-04-16 00:53:38,661 INFO [Listener at localhost/33477] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-04-16 00:53:38,661 INFO [Listener at localhost/33477] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-04-16 00:53:38,661 INFO [Listener at localhost/33477] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-04-16 00:53:38,662 INFO [Listener at localhost/33477] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-04-16 00:53:38,830 INFO [Listener at localhost/33477] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.MasterService, hbase.pb.RegionServerStatusService, hbase.pb.LockService, hbase.pb.HbckService, hbase.pb.ClientMetaService, hbase.pb.ClientService, hbase.pb.AdminService 2023-04-16 00:53:38,928 DEBUG [Listener at localhost/33477] util.ClassSize(228): Using Unsafe to estimate memory layout 2023-04-16 00:53:39,020 INFO [Listener at localhost/33477] ipc.NettyRpcServer(120): Bind to /172.31.14.131:38795 2023-04-16 00:53:39,030 INFO [Listener at localhost/33477] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-16 00:53:39,032 INFO [Listener at localhost/33477] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-16 00:53:39,052 INFO [Listener at localhost/33477] zookeeper.RecoverableZooKeeper(93): Process identifier=master:38795 connecting to ZooKeeper ensemble=127.0.0.1:53150 2023-04-16 00:53:39,093 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:387950x0, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-04-16 00:53:39,098 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): master:38795-0x1018a7584b80000 connected 2023-04-16 00:53:39,132 DEBUG [Listener at localhost/33477] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-04-16 00:53:39,133 DEBUG [Listener at localhost/33477] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-04-16 00:53:39,136 DEBUG [Listener at localhost/33477] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-04-16 00:53:39,144 DEBUG [Listener at localhost/33477] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=38795 2023-04-16 00:53:39,144 DEBUG [Listener at localhost/33477] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=38795 2023-04-16 00:53:39,145 DEBUG [Listener at localhost/33477] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=38795 2023-04-16 00:53:39,145 DEBUG [Listener at localhost/33477] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=38795 2023-04-16 00:53:39,145 DEBUG [Listener at localhost/33477] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=38795 2023-04-16 00:53:39,151 INFO [Listener at localhost/33477] master.HMaster(444): hbase.rootdir=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6, hbase.cluster.distributed=false 2023-04-16 00:53:39,218 INFO [Listener at localhost/33477] client.ConnectionUtils(127): regionserver/jenkins-hbase4:0 server-side Connection retries=45 2023-04-16 00:53:39,218 INFO [Listener at localhost/33477] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-04-16 00:53:39,218 INFO [Listener at localhost/33477] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-04-16 00:53:39,218 INFO [Listener at localhost/33477] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-04-16 00:53:39,219 INFO [Listener at localhost/33477] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-04-16 00:53:39,219 INFO [Listener at localhost/33477] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-04-16 00:53:39,223 INFO [Listener at localhost/33477] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-04-16 00:53:39,227 INFO [Listener at localhost/33477] ipc.NettyRpcServer(120): Bind to /172.31.14.131:34975 2023-04-16 00:53:39,229 INFO [Listener at localhost/33477] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-04-16 00:53:39,235 DEBUG [Listener at localhost/33477] mob.MobFileCache(120): MobFileCache enabled with cacheSize=1000, evictPeriods=3600sec, evictRemainRatio=0.5 2023-04-16 00:53:39,236 INFO [Listener at localhost/33477] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-16 00:53:39,239 INFO [Listener at localhost/33477] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-16 00:53:39,240 INFO [Listener at localhost/33477] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:34975 connecting to ZooKeeper ensemble=127.0.0.1:53150 2023-04-16 00:53:39,244 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:349750x0, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-04-16 00:53:39,245 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): regionserver:34975-0x1018a7584b80001 connected 2023-04-16 00:53:39,245 DEBUG [Listener at localhost/33477] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-04-16 00:53:39,246 DEBUG [Listener at localhost/33477] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-04-16 00:53:39,247 DEBUG [Listener at localhost/33477] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-04-16 00:53:39,248 DEBUG [Listener at localhost/33477] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=34975 2023-04-16 00:53:39,248 DEBUG [Listener at localhost/33477] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=34975 2023-04-16 00:53:39,252 DEBUG [Listener at localhost/33477] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=34975 2023-04-16 00:53:39,253 DEBUG [Listener at localhost/33477] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=34975 2023-04-16 00:53:39,253 DEBUG [Listener at localhost/33477] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=34975 2023-04-16 00:53:39,255 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(2168): Adding backup master ZNode /hbase/backup-masters/jenkins-hbase4.apache.org,38795,1681606417971 2023-04-16 00:53:39,264 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-04-16 00:53:39,265 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/backup-masters/jenkins-hbase4.apache.org,38795,1681606417971 2023-04-16 00:53:39,282 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-04-16 00:53:39,282 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-04-16 00:53:39,283 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-16 00:53:39,284 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-04-16 00:53:39,285 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ActiveMasterManager(227): Deleting ZNode for /hbase/backup-masters/jenkins-hbase4.apache.org,38795,1681606417971 from backup master directory 2023-04-16 00:53:39,285 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-04-16 00:53:39,288 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/backup-masters/jenkins-hbase4.apache.org,38795,1681606417971 2023-04-16 00:53:39,288 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-04-16 00:53:39,289 WARN [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-04-16 00:53:39,289 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ActiveMasterManager(237): Registered as active master=jenkins-hbase4.apache.org,38795,1681606417971 2023-04-16 00:53:39,291 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.ChunkCreator(497): Allocating data MemStoreChunkPool with chunk size 2 MB, max count 352, initial count 0 2023-04-16 00:53:39,293 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.ChunkCreator(497): Allocating index MemStoreChunkPool with chunk size 204.80 KB, max count 391, initial count 0 2023-04-16 00:53:39,375 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] util.FSUtils(620): Created cluster ID file at hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/hbase.id with ID: be39b9c3-e7a2-4cdf-8f0c-28a812c3a4a4 2023-04-16 00:53:39,417 INFO [master/jenkins-hbase4:0:becomeActiveMaster] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-16 00:53:39,431 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-16 00:53:39,471 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ReadOnlyZKClient(139): Connect 0x0e946445 to 127.0.0.1:53150 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-16 00:53:39,505 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@48c8211a, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-16 00:53:39,529 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegion(309): Create or load local region for table 'master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-16 00:53:39,531 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(132): Injected flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000 2023-04-16 00:53:39,552 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(264): ClientProtocol::create wrong number of arguments, should be hadoop 3.2 or below 2023-04-16 00:53:39,552 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(270): ClientProtocol::create wrong number of arguments, should be hadoop 2.x 2023-04-16 00:53:39,554 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(279): can not find SHOULD_REPLICATE flag, should be hadoop 2.x java.lang.IllegalArgumentException: No enum constant org.apache.hadoop.fs.CreateFlag.SHOULD_REPLICATE at java.lang.Enum.valueOf(Enum.java:238) at org.apache.hadoop.fs.CreateFlag.valueOf(CreateFlag.java:63) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.loadShouldReplicateFlag(FanOutOneBlockAsyncDFSOutputHelper.java:277) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.(FanOutOneBlockAsyncDFSOutputHelper.java:304) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.load(AsyncFSWALProvider.java:139) at org.apache.hadoop.hbase.wal.WALFactory.getProviderClass(WALFactory.java:135) at org.apache.hadoop.hbase.wal.WALFactory.getProvider(WALFactory.java:175) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:202) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:182) at org.apache.hadoop.hbase.master.region.MasterRegion.create(MasterRegion.java:339) at org.apache.hadoop.hbase.master.region.MasterRegionFactory.create(MasterRegionFactory.java:104) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:855) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2193) at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:528) at java.lang.Thread.run(Thread.java:750) 2023-04-16 00:53:39,558 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(243): No decryptEncryptedDataEncryptionKey method in DFSClient, should be hadoop version with HDFS-12396 java.lang.NoSuchMethodException: org.apache.hadoop.hdfs.DFSClient.decryptEncryptedDataEncryptionKey(org.apache.hadoop.fs.FileEncryptionInfo) at java.lang.Class.getDeclaredMethod(Class.java:2130) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.createTransparentCryptoHelperWithoutHDFS12396(FanOutOneBlockAsyncDFSOutputSaslHelper.java:182) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.createTransparentCryptoHelper(FanOutOneBlockAsyncDFSOutputSaslHelper.java:241) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.(FanOutOneBlockAsyncDFSOutputSaslHelper.java:252) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.load(AsyncFSWALProvider.java:140) at org.apache.hadoop.hbase.wal.WALFactory.getProviderClass(WALFactory.java:135) at org.apache.hadoop.hbase.wal.WALFactory.getProvider(WALFactory.java:175) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:202) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:182) at org.apache.hadoop.hbase.master.region.MasterRegion.create(MasterRegion.java:339) at org.apache.hadoop.hbase.master.region.MasterRegionFactory.create(MasterRegionFactory.java:104) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:855) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2193) at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:528) at java.lang.Thread.run(Thread.java:750) 2023-04-16 00:53:39,559 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-04-16 00:53:39,593 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7693): Creating {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, under table dir hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/MasterData/data/master/store-tmp 2023-04-16 00:53:39,627 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(866): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-16 00:53:39,627 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1604): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-04-16 00:53:39,627 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1626): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-16 00:53:39,627 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1647): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-16 00:53:39,627 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1714): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-04-16 00:53:39,627 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1724): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-16 00:53:39,627 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1838): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-16 00:53:39,627 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1558): Region close journal for 1595e783b53d99cd5eef43b6debb2682: 2023-04-16 00:53:39,629 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegion(191): WALDir=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/MasterData/WALs/jenkins-hbase4.apache.org,38795,1681606417971 2023-04-16 00:53:39,648 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C38795%2C1681606417971, suffix=, logDir=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/MasterData/WALs/jenkins-hbase4.apache.org,38795,1681606417971, archiveDir=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/MasterData/oldWALs, maxLogs=10 2023-04-16 00:53:39,700 DEBUG [RS-EventLoopGroup-3-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:45865,DS-b52c8e1c-4578-4386-8b7b-56528cefda69,DISK] 2023-04-16 00:53:39,710 DEBUG [RS-EventLoopGroup-3-2] asyncfs.ProtobufDecoder(123): Hadoop 3.2 and below use unshaded protobuf. java.lang.ClassNotFoundException: org.apache.hadoop.thirdparty.protobuf.MessageLite at java.net.URLClassLoader.findClass(URLClassLoader.java:387) at java.lang.ClassLoader.loadClass(ClassLoader.java:418) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352) at java.lang.ClassLoader.loadClass(ClassLoader.java:351) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.io.asyncfs.ProtobufDecoder.(ProtobufDecoder.java:118) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.processWriteBlockResponse(FanOutOneBlockAsyncDFSOutputHelper.java:340) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$100(FanOutOneBlockAsyncDFSOutputHelper.java:112) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$4.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:424) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:557) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.addListener(DefaultPromise.java:185) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.initialize(FanOutOneBlockAsyncDFSOutputHelper.java:418) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$300(FanOutOneBlockAsyncDFSOutputHelper.java:112) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$5.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:476) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$5.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:471) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:583) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:559) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:636) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setSuccess0(DefaultPromise.java:625) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:105) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:84) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.fulfillConnectPromise(AbstractEpollChannel.java:653) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:691) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:567) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:489) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:397) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-16 00:53:39,781 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/MasterData/WALs/jenkins-hbase4.apache.org,38795,1681606417971/jenkins-hbase4.apache.org%2C38795%2C1681606417971.1681606419658 2023-04-16 00:53:39,781 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:45865,DS-b52c8e1c-4578-4386-8b7b-56528cefda69,DISK]] 2023-04-16 00:53:39,782 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7854): Opening region: {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''} 2023-04-16 00:53:39,783 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(866): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-16 00:53:39,786 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7894): checking encryption for 1595e783b53d99cd5eef43b6debb2682 2023-04-16 00:53:39,788 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7897): checking classloading for 1595e783b53d99cd5eef43b6debb2682 2023-04-16 00:53:39,859 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family proc of region 1595e783b53d99cd5eef43b6debb2682 2023-04-16 00:53:39,867 DEBUG [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc 2023-04-16 00:53:39,895 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1595e783b53d99cd5eef43b6debb2682 columnFamilyName proc 2023-04-16 00:53:39,907 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(310): Store=1595e783b53d99cd5eef43b6debb2682/proc, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-16 00:53:39,916 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-04-16 00:53:39,918 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-04-16 00:53:39,934 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1055): writing seq id for 1595e783b53d99cd5eef43b6debb2682 2023-04-16 00:53:39,939 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-16 00:53:39,940 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1072): Opened 1595e783b53d99cd5eef43b6debb2682; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=64869435, jitterRate=-0.03337009251117706}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-04-16 00:53:39,940 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(965): Region open journal for 1595e783b53d99cd5eef43b6debb2682: 2023-04-16 00:53:39,944 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(122): Constructor flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000, compactMin=4 2023-04-16 00:53:39,965 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=5 2023-04-16 00:53:39,965 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(562): Starting 5 core workers (bigger of cpus/4 or 16) with max (burst) worker count=50 2023-04-16 00:53:39,967 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-04-16 00:53:39,969 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 1 msec 2023-04-16 00:53:40,004 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 34 msec 2023-04-16 00:53:40,004 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.RemoteProcedureDispatcher(95): Instantiated, coreThreads=3 (allowCoreThreadTimeOut=true), queueMaxSize=32, operationDelay=150 2023-04-16 00:53:40,031 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(253): hbase:meta replica znodes: [] 2023-04-16 00:53:40,036 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.RegionServerTracker(124): Starting RegionServerTracker; 0 have existing ServerCrashProcedures, 0 possibly 'live' servers, and 0 'splitting'. 2023-04-16 00:53:40,066 INFO [master/jenkins-hbase4:0:becomeActiveMaster] balancer.BaseLoadBalancer(1082): slop=0.001, systemTablesOnMaster=false 2023-04-16 00:53:40,072 INFO [master/jenkins-hbase4:0:becomeActiveMaster] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, ServerLocalityCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-04-16 00:53:40,074 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/balancer 2023-04-16 00:53:40,081 INFO [master/jenkins-hbase4:0:becomeActiveMaster] normalizer.RegionNormalizerWorker(118): Normalizer rate limit set to unlimited 2023-04-16 00:53:40,087 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/normalizer 2023-04-16 00:53:40,090 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-16 00:53:40,092 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/split 2023-04-16 00:53:40,093 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/merge 2023-04-16 00:53:40,107 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/snapshot-cleanup 2023-04-16 00:53:40,112 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-04-16 00:53:40,112 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-04-16 00:53:40,113 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-16 00:53:40,113 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(744): Active/primary master=jenkins-hbase4.apache.org,38795,1681606417971, sessionid=0x1018a7584b80000, setting cluster-up flag (Was=false) 2023-04-16 00:53:40,130 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-16 00:53:40,140 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/flush-table-proc/acquired, /hbase/flush-table-proc/reached, /hbase/flush-table-proc/abort 2023-04-16 00:53:40,142 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase4.apache.org,38795,1681606417971 2023-04-16 00:53:40,147 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-16 00:53:40,158 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/online-snapshot/acquired, /hbase/online-snapshot/reached, /hbase/online-snapshot/abort 2023-04-16 00:53:40,162 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase4.apache.org,38795,1681606417971 2023-04-16 00:53:40,165 WARN [master/jenkins-hbase4:0:becomeActiveMaster] snapshot.SnapshotManager(302): Couldn't delete working snapshot directory: hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/.hbase-snapshot/.tmp 2023-04-16 00:53:40,257 INFO [RS:0;jenkins-hbase4:34975] regionserver.HRegionServer(952): ClusterId : be39b9c3-e7a2-4cdf-8f0c-28a812c3a4a4 2023-04-16 00:53:40,263 DEBUG [RS:0;jenkins-hbase4:34975] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-04-16 00:53:40,269 DEBUG [RS:0;jenkins-hbase4:34975] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-04-16 00:53:40,269 DEBUG [RS:0;jenkins-hbase4:34975] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-04-16 00:53:40,272 DEBUG [RS:0;jenkins-hbase4:34975] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-04-16 00:53:40,273 DEBUG [RS:0;jenkins-hbase4:34975] zookeeper.ReadOnlyZKClient(139): Connect 0x62be487e to 127.0.0.1:53150 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-16 00:53:40,284 DEBUG [RS:0;jenkins-hbase4:34975] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6c79ba59, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-16 00:53:40,285 DEBUG [RS:0;jenkins-hbase4:34975] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6846230e, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-04-16 00:53:40,310 DEBUG [RS:0;jenkins-hbase4:34975] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:0;jenkins-hbase4:34975 2023-04-16 00:53:40,314 INFO [RS:0;jenkins-hbase4:34975] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-04-16 00:53:40,314 INFO [RS:0;jenkins-hbase4:34975] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-04-16 00:53:40,314 DEBUG [RS:0;jenkins-hbase4:34975] regionserver.HRegionServer(1023): About to register with Master. 2023-04-16 00:53:40,316 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT; InitMetaProcedure table=hbase:meta 2023-04-16 00:53:40,316 INFO [RS:0;jenkins-hbase4:34975] regionserver.HRegionServer(2810): reportForDuty to master=jenkins-hbase4.apache.org,38795,1681606417971 with isa=jenkins-hbase4.apache.org/172.31.14.131:34975, startcode=1681606419217 2023-04-16 00:53:40,329 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_OPEN_REGION-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-04-16 00:53:40,329 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_CLOSE_REGION-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-04-16 00:53:40,329 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SERVER_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-04-16 00:53:40,329 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_META_SERVER_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-04-16 00:53:40,329 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=M_LOG_REPLAY_OPS-master/jenkins-hbase4:0, corePoolSize=10, maxPoolSize=10 2023-04-16 00:53:40,330 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-16 00:53:40,330 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_MERGE_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-04-16 00:53:40,330 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_TABLE_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-16 00:53:40,332 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1681606450332 2023-04-16 00:53:40,335 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.DirScanPool(70): log_cleaner Cleaner pool size is 1 2023-04-16 00:53:40,337 DEBUG [RS:0;jenkins-hbase4:34975] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-04-16 00:53:40,339 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT, locked=true; InitMetaProcedure table=hbase:meta 2023-04-16 00:53:40,339 INFO [PEWorker-1] procedure.InitMetaProcedure(71): BOOTSTRAP: creating hbase:meta region 2023-04-16 00:53:40,349 INFO [PEWorker-1] util.FSTableDescriptors(128): Creating new hbase:meta table descriptor 'hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-04-16 00:53:40,349 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveLogCleaner 2023-04-16 00:53:40,357 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner 2023-04-16 00:53:40,358 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreWALCleaner 2023-04-16 00:53:40,358 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveProcedureWALCleaner 2023-04-16 00:53:40,358 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.LogCleaner(148): Creating 1 old WALs cleaner threads 2023-04-16 00:53:40,358 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=LogsCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-04-16 00:53:40,360 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.DirScanPool(70): hfile_cleaner Cleaner pool size is 2 2023-04-16 00:53:40,362 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreHFileCleaner 2023-04-16 00:53:40,363 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.HFileLinkCleaner 2023-04-16 00:53:40,367 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.snapshot.SnapshotHFileCleaner 2023-04-16 00:53:40,368 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveHFileCleaner 2023-04-16 00:53:40,376 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.HFileCleaner(242): Starting for large file=Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1681606420376,5,FailOnTimeoutGroup] 2023-04-16 00:53:40,379 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.HFileCleaner(257): Starting for small files=Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1681606420379,5,FailOnTimeoutGroup] 2023-04-16 00:53:40,379 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HFileCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-04-16 00:53:40,380 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1461): Reopening regions with very high storeFileRefCount is disabled. Provide threshold value > 0 for hbase.regions.recovery.store.file.ref.count to enable it. 2023-04-16 00:53:40,382 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=ReplicationBarrierCleaner, period=43200000, unit=MILLISECONDS is enabled. 2023-04-16 00:53:40,383 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=SnapshotCleaner, period=1800000, unit=MILLISECONDS is enabled. 2023-04-16 00:53:40,403 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:38469, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins.hfs.0 (auth:SIMPLE), service=RegionServerStatusService 2023-04-16 00:53:40,413 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.ServerManager(394): Registering regionserver=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:40,429 DEBUG [RS:0;jenkins-hbase4:34975] regionserver.HRegionServer(1596): Config from master: hbase.rootdir=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6 2023-04-16 00:53:40,429 DEBUG [RS:0;jenkins-hbase4:34975] regionserver.HRegionServer(1596): Config from master: fs.defaultFS=hdfs://localhost:43665 2023-04-16 00:53:40,429 DEBUG [RS:0;jenkins-hbase4:34975] regionserver.HRegionServer(1596): Config from master: hbase.master.info.port=-1 2023-04-16 00:53:40,435 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-04-16 00:53:40,436 DEBUG [RS:0;jenkins-hbase4:34975] zookeeper.ZKUtil(162): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:40,436 WARN [RS:0;jenkins-hbase4:34975] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-04-16 00:53:40,437 INFO [RS:0;jenkins-hbase4:34975] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-04-16 00:53:40,437 DEBUG [RS:0;jenkins-hbase4:34975] regionserver.HRegionServer(1947): logDir=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/WALs/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:40,438 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase4.apache.org,34975,1681606419217] 2023-04-16 00:53:40,446 DEBUG [RS:0;jenkins-hbase4:34975] zookeeper.ZKUtil(162): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:40,459 DEBUG [RS:0;jenkins-hbase4:34975] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-04-16 00:53:40,470 INFO [RS:0;jenkins-hbase4:34975] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-04-16 00:53:40,496 INFO [RS:0;jenkins-hbase4:34975] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-04-16 00:53:40,501 INFO [RS:0;jenkins-hbase4:34975] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-04-16 00:53:40,501 INFO [RS:0;jenkins-hbase4:34975] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-04-16 00:53:40,502 INFO [RS:0;jenkins-hbase4:34975] regionserver.HRegionServer$CompactionChecker(1838): CompactionChecker runs every PT1S 2023-04-16 00:53:40,511 INFO [RS:0;jenkins-hbase4:34975] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-04-16 00:53:40,512 DEBUG [RS:0;jenkins-hbase4:34975] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-16 00:53:40,512 DEBUG [RS:0;jenkins-hbase4:34975] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-16 00:53:40,513 DEBUG [RS:0;jenkins-hbase4:34975] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-16 00:53:40,513 DEBUG [RS:0;jenkins-hbase4:34975] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-16 00:53:40,513 DEBUG [RS:0;jenkins-hbase4:34975] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-16 00:53:40,513 DEBUG [RS:0;jenkins-hbase4:34975] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-04-16 00:53:40,513 DEBUG [RS:0;jenkins-hbase4:34975] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-16 00:53:40,514 DEBUG [RS:0;jenkins-hbase4:34975] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-16 00:53:40,514 DEBUG [RS:0;jenkins-hbase4:34975] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-16 00:53:40,514 DEBUG [RS:0;jenkins-hbase4:34975] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-16 00:53:40,515 INFO [RS:0;jenkins-hbase4:34975] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-04-16 00:53:40,515 INFO [RS:0;jenkins-hbase4:34975] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-04-16 00:53:40,515 INFO [RS:0;jenkins-hbase4:34975] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-04-16 00:53:40,534 INFO [RS:0;jenkins-hbase4:34975] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-04-16 00:53:40,536 INFO [RS:0;jenkins-hbase4:34975] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,34975,1681606419217-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-04-16 00:53:40,553 INFO [RS:0;jenkins-hbase4:34975] regionserver.Replication(203): jenkins-hbase4.apache.org,34975,1681606419217 started 2023-04-16 00:53:40,553 INFO [RS:0;jenkins-hbase4:34975] regionserver.HRegionServer(1638): Serving as jenkins-hbase4.apache.org,34975,1681606419217, RpcServer on jenkins-hbase4.apache.org/172.31.14.131:34975, sessionid=0x1018a7584b80001 2023-04-16 00:53:40,553 DEBUG [RS:0;jenkins-hbase4:34975] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-04-16 00:53:40,553 DEBUG [RS:0;jenkins-hbase4:34975] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:40,553 DEBUG [RS:0;jenkins-hbase4:34975] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,34975,1681606419217' 2023-04-16 00:53:40,553 DEBUG [RS:0;jenkins-hbase4:34975] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:53:40,554 DEBUG [RS:0;jenkins-hbase4:34975] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:53:40,555 DEBUG [RS:0;jenkins-hbase4:34975] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-04-16 00:53:40,555 DEBUG [RS:0;jenkins-hbase4:34975] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-04-16 00:53:40,555 DEBUG [RS:0;jenkins-hbase4:34975] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:40,555 DEBUG [RS:0;jenkins-hbase4:34975] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,34975,1681606419217' 2023-04-16 00:53:40,555 DEBUG [RS:0;jenkins-hbase4:34975] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-04-16 00:53:40,555 DEBUG [RS:0;jenkins-hbase4:34975] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-04-16 00:53:40,556 DEBUG [RS:0;jenkins-hbase4:34975] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-04-16 00:53:40,556 INFO [RS:0;jenkins-hbase4:34975] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-04-16 00:53:40,556 INFO [RS:0;jenkins-hbase4:34975] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-04-16 00:53:40,671 INFO [RS:0;jenkins-hbase4:34975] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C34975%2C1681606419217, suffix=, logDir=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/WALs/jenkins-hbase4.apache.org,34975,1681606419217, archiveDir=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/oldWALs, maxLogs=32 2023-04-16 00:53:40,696 DEBUG [RS-EventLoopGroup-3-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:45865,DS-b52c8e1c-4578-4386-8b7b-56528cefda69,DISK] 2023-04-16 00:53:40,703 INFO [RS:0;jenkins-hbase4:34975] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/WALs/jenkins-hbase4.apache.org,34975,1681606419217/jenkins-hbase4.apache.org%2C34975%2C1681606419217.1681606420674 2023-04-16 00:53:40,704 DEBUG [RS:0;jenkins-hbase4:34975] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:45865,DS-b52c8e1c-4578-4386-8b7b-56528cefda69,DISK]] 2023-04-16 00:53:40,790 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-04-16 00:53:40,791 INFO [PEWorker-1] util.FSTableDescriptors(135): Updated hbase:meta table descriptor to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-04-16 00:53:40,791 INFO [PEWorker-1] regionserver.HRegion(7675): creating {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6 2023-04-16 00:53:41,209 DEBUG [PEWorker-1] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-16 00:53:41,211 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-04-16 00:53:41,214 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/hbase/meta/1588230740/info 2023-04-16 00:53:41,215 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-04-16 00:53:41,216 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-16 00:53:41,216 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-04-16 00:53:41,219 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/hbase/meta/1588230740/rep_barrier 2023-04-16 00:53:41,220 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-04-16 00:53:41,220 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-16 00:53:41,221 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-04-16 00:53:41,223 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/hbase/meta/1588230740/table 2023-04-16 00:53:41,223 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-04-16 00:53:41,224 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-16 00:53:41,226 DEBUG [PEWorker-1] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/hbase/meta/1588230740 2023-04-16 00:53:41,227 DEBUG [PEWorker-1] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/hbase/meta/1588230740 2023-04-16 00:53:41,231 DEBUG [PEWorker-1] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-16 00:53:41,233 DEBUG [PEWorker-1] regionserver.HRegion(1055): writing seq id for 1588230740 2023-04-16 00:53:41,236 DEBUG [PEWorker-1] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/hbase/meta/1588230740/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-16 00:53:41,238 INFO [PEWorker-1] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=73997034, jitterRate=0.10264173150062561}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-16 00:53:41,238 DEBUG [PEWorker-1] regionserver.HRegion(965): Region open journal for 1588230740: 2023-04-16 00:53:41,238 DEBUG [PEWorker-1] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-04-16 00:53:41,238 INFO [PEWorker-1] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 2023-04-16 00:53:41,238 DEBUG [PEWorker-1] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-04-16 00:53:41,238 DEBUG [PEWorker-1] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-04-16 00:53:41,238 DEBUG [PEWorker-1] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 2023-04-16 00:53:41,240 INFO [PEWorker-1] regionserver.HRegion(1838): Closed hbase:meta,,1.1588230740 2023-04-16 00:53:41,240 DEBUG [PEWorker-1] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-04-16 00:53:41,246 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_ASSIGN_META, locked=true; InitMetaProcedure table=hbase:meta 2023-04-16 00:53:41,246 INFO [PEWorker-1] procedure.InitMetaProcedure(103): Going to assign meta 2023-04-16 00:53:41,256 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN}] 2023-04-16 00:53:41,271 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN 2023-04-16 00:53:41,273 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN; state=OFFLINE, location=null; forceNewPlan=false, retain=false 2023-04-16 00:53:41,425 DEBUG [jenkins-hbase4:38795] assignment.AssignmentManager(2176): Processing assignQueue; systemServersCount=1, allServersCount=1 2023-04-16 00:53:41,428 INFO [PEWorker-3] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase4.apache.org,34975,1681606419217, state=OPENING 2023-04-16 00:53:41,435 DEBUG [PEWorker-3] zookeeper.MetaTableLocator(240): hbase:meta region location doesn't exist, create it 2023-04-16 00:53:41,437 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-16 00:53:41,438 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-04-16 00:53:41,440 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=3, ppid=2, state=RUNNABLE; OpenRegionProcedure 1588230740, server=jenkins-hbase4.apache.org,34975,1681606419217}] 2023-04-16 00:53:41,623 DEBUG [RSProcedureDispatcher-pool-0] master.ServerManager(712): New admin connection to jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:41,625 DEBUG [RSProcedureDispatcher-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-04-16 00:53:41,629 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:43014, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-04-16 00:53:41,640 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open hbase:meta,,1.1588230740 2023-04-16 00:53:41,640 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-04-16 00:53:41,643 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C34975%2C1681606419217.meta, suffix=.meta, logDir=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/WALs/jenkins-hbase4.apache.org,34975,1681606419217, archiveDir=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/oldWALs, maxLogs=32 2023-04-16 00:53:41,661 DEBUG [RS-EventLoopGroup-3-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:45865,DS-b52c8e1c-4578-4386-8b7b-56528cefda69,DISK] 2023-04-16 00:53:41,666 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/WALs/jenkins-hbase4.apache.org,34975,1681606419217/jenkins-hbase4.apache.org%2C34975%2C1681606419217.meta.1681606421645.meta 2023-04-16 00:53:41,666 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:45865,DS-b52c8e1c-4578-4386-8b7b-56528cefda69,DISK]] 2023-04-16 00:53:41,666 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''} 2023-04-16 00:53:41,668 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(215): Loading coprocessor class org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint with path null and priority 536870911 2023-04-16 00:53:41,686 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(8550): Registered coprocessor service: region=hbase:meta,,1 service=MultiRowMutationService 2023-04-16 00:53:41,691 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.RegionCoprocessorHost(393): Loaded coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint from HTD of hbase:meta successfully. 2023-04-16 00:53:41,696 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table meta 1588230740 2023-04-16 00:53:41,696 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-16 00:53:41,696 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 1588230740 2023-04-16 00:53:41,696 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 1588230740 2023-04-16 00:53:41,699 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-04-16 00:53:41,701 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/hbase/meta/1588230740/info 2023-04-16 00:53:41,701 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/hbase/meta/1588230740/info 2023-04-16 00:53:41,702 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-04-16 00:53:41,703 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-16 00:53:41,703 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-04-16 00:53:41,705 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/hbase/meta/1588230740/rep_barrier 2023-04-16 00:53:41,705 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/hbase/meta/1588230740/rep_barrier 2023-04-16 00:53:41,706 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-04-16 00:53:41,707 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-16 00:53:41,707 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-04-16 00:53:41,708 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/hbase/meta/1588230740/table 2023-04-16 00:53:41,708 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/hbase/meta/1588230740/table 2023-04-16 00:53:41,709 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-04-16 00:53:41,710 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-16 00:53:41,712 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/hbase/meta/1588230740 2023-04-16 00:53:41,714 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/hbase/meta/1588230740 2023-04-16 00:53:41,718 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-16 00:53:41,721 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 1588230740 2023-04-16 00:53:41,722 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=74632449, jitterRate=0.11211015284061432}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-16 00:53:41,722 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 1588230740: 2023-04-16 00:53:41,732 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for hbase:meta,,1.1588230740, pid=3, masterSystemTime=1681606421616 2023-04-16 00:53:41,751 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for hbase:meta,,1.1588230740 2023-04-16 00:53:41,752 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened hbase:meta,,1.1588230740 2023-04-16 00:53:41,752 INFO [PEWorker-5] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase4.apache.org,34975,1681606419217, state=OPEN 2023-04-16 00:53:41,769 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2023-04-16 00:53:41,769 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-04-16 00:53:41,774 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=3, resume processing ppid=2 2023-04-16 00:53:41,774 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=3, ppid=2, state=SUCCESS; OpenRegionProcedure 1588230740, server=jenkins-hbase4.apache.org,34975,1681606419217 in 329 msec 2023-04-16 00:53:41,779 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=2, resume processing ppid=1 2023-04-16 00:53:41,780 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=2, ppid=1, state=SUCCESS; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN in 520 msec 2023-04-16 00:53:41,784 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta in 1.5540 sec 2023-04-16 00:53:41,784 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(953): Wait for region servers to report in: status=null, state=RUNNING, startTime=1681606421784, completionTime=-1 2023-04-16 00:53:41,785 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ServerManager(821): Finished waiting on RegionServer count=1; waited=0ms, expected min=1 server(s), max=1 server(s), master is running 2023-04-16 00:53:41,785 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1517): Joining cluster... 2023-04-16 00:53:41,847 DEBUG [hconnection-0x4a6b0192-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-16 00:53:41,851 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:43018, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-16 00:53:41,869 INFO [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1529): Number of RegionServers=1 2023-04-16 00:53:41,869 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1681606481869 2023-04-16 00:53:41,870 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1681606541870 2023-04-16 00:53:41,870 INFO [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1536): Joined the cluster in 84 msec 2023-04-16 00:53:41,964 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,38795,1681606417971-ClusterStatusChore, period=60000, unit=MILLISECONDS is enabled. 2023-04-16 00:53:41,965 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,38795,1681606417971-BalancerChore, period=300000, unit=MILLISECONDS is enabled. 2023-04-16 00:53:41,965 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,38795,1681606417971-RegionNormalizerChore, period=300000, unit=MILLISECONDS is enabled. 2023-04-16 00:53:41,967 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=CatalogJanitor-jenkins-hbase4:38795, period=300000, unit=MILLISECONDS is enabled. 2023-04-16 00:53:41,967 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HbckChore-, period=3600000, unit=MILLISECONDS is enabled. 2023-04-16 00:53:41,974 DEBUG [master/jenkins-hbase4:0.Chore.1] janitor.CatalogJanitor(175): 2023-04-16 00:53:41,981 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.TableNamespaceManager(92): Namespace table not found. Creating... 2023-04-16 00:53:41,983 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(2148): Client=null/null create 'hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-04-16 00:53:41,991 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=hbase:namespace 2023-04-16 00:53:41,993 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_PRE_OPERATION 2023-04-16 00:53:41,996 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-16 00:53:42,017 DEBUG [HFileArchiver-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/.tmp/data/hbase/namespace/bebb4c20d3b4786f542e659d1c13e314 2023-04-16 00:53:42,019 DEBUG [HFileArchiver-1] backup.HFileArchiver(153): Directory hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/.tmp/data/hbase/namespace/bebb4c20d3b4786f542e659d1c13e314 empty. 2023-04-16 00:53:42,020 DEBUG [HFileArchiver-1] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/.tmp/data/hbase/namespace/bebb4c20d3b4786f542e659d1c13e314 2023-04-16 00:53:42,020 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(328): Archived hbase:namespace regions 2023-04-16 00:53:42,443 DEBUG [PEWorker-3] util.FSTableDescriptors(570): Wrote into hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/.tmp/data/hbase/namespace/.tabledesc/.tableinfo.0000000001 2023-04-16 00:53:42,446 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(7675): creating {ENCODED => bebb4c20d3b4786f542e659d1c13e314, NAME => 'hbase:namespace,,1681606421982.bebb4c20d3b4786f542e659d1c13e314.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/.tmp 2023-04-16 00:53:42,860 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1681606421982.bebb4c20d3b4786f542e659d1c13e314.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-16 00:53:42,860 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1604): Closing bebb4c20d3b4786f542e659d1c13e314, disabling compactions & flushes 2023-04-16 00:53:42,861 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1681606421982.bebb4c20d3b4786f542e659d1c13e314. 2023-04-16 00:53:42,861 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1681606421982.bebb4c20d3b4786f542e659d1c13e314. 2023-04-16 00:53:42,861 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1681606421982.bebb4c20d3b4786f542e659d1c13e314. after waiting 0 ms 2023-04-16 00:53:42,861 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1681606421982.bebb4c20d3b4786f542e659d1c13e314. 2023-04-16 00:53:42,861 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1838): Closed hbase:namespace,,1681606421982.bebb4c20d3b4786f542e659d1c13e314. 2023-04-16 00:53:42,861 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1558): Region close journal for bebb4c20d3b4786f542e659d1c13e314: 2023-04-16 00:53:42,866 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ADD_TO_META 2023-04-16 00:53:42,881 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"hbase:namespace,,1681606421982.bebb4c20d3b4786f542e659d1c13e314.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1681606422869"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681606422869"}]},"ts":"1681606422869"} 2023-04-16 00:53:42,909 INFO [PEWorker-3] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-16 00:53:42,911 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-16 00:53:42,915 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681606422911"}]},"ts":"1681606422911"} 2023-04-16 00:53:42,920 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLING in hbase:meta 2023-04-16 00:53:42,931 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=bebb4c20d3b4786f542e659d1c13e314, ASSIGN}] 2023-04-16 00:53:42,934 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=bebb4c20d3b4786f542e659d1c13e314, ASSIGN 2023-04-16 00:53:42,936 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:namespace, region=bebb4c20d3b4786f542e659d1c13e314, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,34975,1681606419217; forceNewPlan=false, retain=false 2023-04-16 00:53:43,087 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=bebb4c20d3b4786f542e659d1c13e314, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:43,088 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"hbase:namespace,,1681606421982.bebb4c20d3b4786f542e659d1c13e314.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1681606423087"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681606423087"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681606423087"}]},"ts":"1681606423087"} 2023-04-16 00:53:43,093 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=6, ppid=5, state=RUNNABLE; OpenRegionProcedure bebb4c20d3b4786f542e659d1c13e314, server=jenkins-hbase4.apache.org,34975,1681606419217}] 2023-04-16 00:53:43,252 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open hbase:namespace,,1681606421982.bebb4c20d3b4786f542e659d1c13e314. 2023-04-16 00:53:43,253 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => bebb4c20d3b4786f542e659d1c13e314, NAME => 'hbase:namespace,,1681606421982.bebb4c20d3b4786f542e659d1c13e314.', STARTKEY => '', ENDKEY => ''} 2023-04-16 00:53:43,254 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table namespace bebb4c20d3b4786f542e659d1c13e314 2023-04-16 00:53:43,254 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1681606421982.bebb4c20d3b4786f542e659d1c13e314.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-16 00:53:43,254 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for bebb4c20d3b4786f542e659d1c13e314 2023-04-16 00:53:43,254 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for bebb4c20d3b4786f542e659d1c13e314 2023-04-16 00:53:43,257 INFO [StoreOpener-bebb4c20d3b4786f542e659d1c13e314-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region bebb4c20d3b4786f542e659d1c13e314 2023-04-16 00:53:43,259 DEBUG [StoreOpener-bebb4c20d3b4786f542e659d1c13e314-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/hbase/namespace/bebb4c20d3b4786f542e659d1c13e314/info 2023-04-16 00:53:43,259 DEBUG [StoreOpener-bebb4c20d3b4786f542e659d1c13e314-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/hbase/namespace/bebb4c20d3b4786f542e659d1c13e314/info 2023-04-16 00:53:43,260 INFO [StoreOpener-bebb4c20d3b4786f542e659d1c13e314-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region bebb4c20d3b4786f542e659d1c13e314 columnFamilyName info 2023-04-16 00:53:43,261 INFO [StoreOpener-bebb4c20d3b4786f542e659d1c13e314-1] regionserver.HStore(310): Store=bebb4c20d3b4786f542e659d1c13e314/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-16 00:53:43,262 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/hbase/namespace/bebb4c20d3b4786f542e659d1c13e314 2023-04-16 00:53:43,263 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/hbase/namespace/bebb4c20d3b4786f542e659d1c13e314 2023-04-16 00:53:43,268 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for bebb4c20d3b4786f542e659d1c13e314 2023-04-16 00:53:43,272 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/hbase/namespace/bebb4c20d3b4786f542e659d1c13e314/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-16 00:53:43,273 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened bebb4c20d3b4786f542e659d1c13e314; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=72886896, jitterRate=0.08609938621520996}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-04-16 00:53:43,273 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for bebb4c20d3b4786f542e659d1c13e314: 2023-04-16 00:53:43,275 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for hbase:namespace,,1681606421982.bebb4c20d3b4786f542e659d1c13e314., pid=6, masterSystemTime=1681606423246 2023-04-16 00:53:43,279 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for hbase:namespace,,1681606421982.bebb4c20d3b4786f542e659d1c13e314. 2023-04-16 00:53:43,279 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened hbase:namespace,,1681606421982.bebb4c20d3b4786f542e659d1c13e314. 2023-04-16 00:53:43,280 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=bebb4c20d3b4786f542e659d1c13e314, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:43,281 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"hbase:namespace,,1681606421982.bebb4c20d3b4786f542e659d1c13e314.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1681606423280"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1681606423280"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1681606423280"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1681606423280"}]},"ts":"1681606423280"} 2023-04-16 00:53:43,288 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=6, resume processing ppid=5 2023-04-16 00:53:43,288 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=6, ppid=5, state=SUCCESS; OpenRegionProcedure bebb4c20d3b4786f542e659d1c13e314, server=jenkins-hbase4.apache.org,34975,1681606419217 in 191 msec 2023-04-16 00:53:43,291 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=5, resume processing ppid=4 2023-04-16 00:53:43,292 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=5, ppid=4, state=SUCCESS; TransitRegionStateProcedure table=hbase:namespace, region=bebb4c20d3b4786f542e659d1c13e314, ASSIGN in 357 msec 2023-04-16 00:53:43,293 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-16 00:53:43,294 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681606423293"}]},"ts":"1681606423293"} 2023-04-16 00:53:43,297 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLED in hbase:meta 2023-04-16 00:53:43,300 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_POST_OPERATION 2023-04-16 00:53:43,303 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace in 1.3170 sec 2023-04-16 00:53:43,395 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/namespace 2023-04-16 00:53:43,397 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/namespace 2023-04-16 00:53:43,397 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-16 00:53:43,439 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=7, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=default 2023-04-16 00:53:43,457 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-04-16 00:53:43,465 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default in 35 msec 2023-04-16 00:53:43,474 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=8, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=hbase 2023-04-16 00:53:43,488 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-04-16 00:53:43,495 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase in 21 msec 2023-04-16 00:53:43,502 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/default 2023-04-16 00:53:43,504 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/hbase 2023-04-16 00:53:43,505 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1083): Master has completed initialization 4.215sec 2023-04-16 00:53:43,506 INFO [master/jenkins-hbase4:0:becomeActiveMaster] quotas.MasterQuotaManager(97): Quota support disabled 2023-04-16 00:53:43,508 INFO [master/jenkins-hbase4:0:becomeActiveMaster] slowlog.SlowLogMasterService(57): Slow/Large requests logging to system table hbase:slowlog is disabled. Quitting. 2023-04-16 00:53:43,508 INFO [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKWatcher(269): not a secure deployment, proceeding 2023-04-16 00:53:43,509 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,38795,1681606417971-ExpiredMobFileCleanerChore, period=86400, unit=SECONDS is enabled. 2023-04-16 00:53:43,510 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,38795,1681606417971-MobCompactionChore, period=604800, unit=SECONDS is enabled. 2023-04-16 00:53:43,520 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1175): Balancer post startup initialization complete, took 0 seconds 2023-04-16 00:53:43,564 DEBUG [Listener at localhost/33477] zookeeper.ReadOnlyZKClient(139): Connect 0x6ed65498 to 127.0.0.1:53150 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-16 00:53:43,569 DEBUG [Listener at localhost/33477] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@f3002be, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-16 00:53:43,582 DEBUG [hconnection-0x14c3948e-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-16 00:53:43,595 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:39966, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-16 00:53:43,606 INFO [Listener at localhost/33477] hbase.HBaseTestingUtility(1145): Minicluster is up; activeMaster=jenkins-hbase4.apache.org,38795,1681606417971 2023-04-16 00:53:43,630 INFO [Listener at localhost/33477] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testMixedAtomicity Thread=206, OpenFileDescriptor=470, MaxFileDescriptor=60000, SystemLoadAverage=227, ProcessCount=171, AvailableMemoryMB=1619 2023-04-16 00:53:43,649 DEBUG [Listener at localhost/33477] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-16 00:53:43,653 INFO [RS-EventLoopGroup-1-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:35154, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-16 00:53:43,664 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-16 00:53:43,668 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-16 00:53:43,671 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure2.ProcedureExecutor(1029): Stored pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-16 00:53:43,673 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-16 00:53:43,676 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-16 00:53:43,678 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 9 2023-04-16 00:53:43,679 DEBUG [HFileArchiver-2] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/.tmp/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba 2023-04-16 00:53:43,680 DEBUG [HFileArchiver-2] backup.HFileArchiver(153): Directory hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/.tmp/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba empty. 2023-04-16 00:53:43,682 DEBUG [HFileArchiver-2] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/.tmp/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba 2023-04-16 00:53:43,682 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-16 00:53:43,689 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-04-16 00:53:43,703 DEBUG [PEWorker-2] util.FSTableDescriptors(570): Wrote into hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-16 00:53:43,705 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => c9039369968c422a52a2c7db83b430ba, NAME => 'TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/.tmp 2023-04-16 00:53:43,723 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-16 00:53:43,723 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing c9039369968c422a52a2c7db83b430ba, disabling compactions & flushes 2023-04-16 00:53:43,723 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. 2023-04-16 00:53:43,723 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. 2023-04-16 00:53:43,723 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. after waiting 0 ms 2023-04-16 00:53:43,723 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. 2023-04-16 00:53:43,723 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. 2023-04-16 00:53:43,723 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for c9039369968c422a52a2c7db83b430ba: 2023-04-16 00:53:43,728 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-16 00:53:43,730 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681606423729"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681606423729"}]},"ts":"1681606423729"} 2023-04-16 00:53:43,732 INFO [PEWorker-2] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-16 00:53:43,734 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-16 00:53:43,734 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681606423734"}]},"ts":"1681606423734"} 2023-04-16 00:53:43,737 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-16 00:53:43,742 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=c9039369968c422a52a2c7db83b430ba, ASSIGN}] 2023-04-16 00:53:43,744 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=c9039369968c422a52a2c7db83b430ba, ASSIGN 2023-04-16 00:53:43,747 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=c9039369968c422a52a2c7db83b430ba, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,34975,1681606419217; forceNewPlan=false, retain=false 2023-04-16 00:53:43,795 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-04-16 00:53:43,899 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=c9039369968c422a52a2c7db83b430ba, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:43,899 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681606423898"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681606423898"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681606423898"}]},"ts":"1681606423898"} 2023-04-16 00:53:43,903 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=11, ppid=10, state=RUNNABLE; OpenRegionProcedure c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217}] 2023-04-16 00:53:43,997 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-04-16 00:53:44,063 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. 2023-04-16 00:53:44,064 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => c9039369968c422a52a2c7db83b430ba, NAME => 'TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba.', STARTKEY => '', ENDKEY => ''} 2023-04-16 00:53:44,064 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees c9039369968c422a52a2c7db83b430ba 2023-04-16 00:53:44,064 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-16 00:53:44,065 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for c9039369968c422a52a2c7db83b430ba 2023-04-16 00:53:44,065 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for c9039369968c422a52a2c7db83b430ba 2023-04-16 00:53:44,067 INFO [StoreOpener-c9039369968c422a52a2c7db83b430ba-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region c9039369968c422a52a2c7db83b430ba 2023-04-16 00:53:44,069 DEBUG [StoreOpener-c9039369968c422a52a2c7db83b430ba-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/A 2023-04-16 00:53:44,069 DEBUG [StoreOpener-c9039369968c422a52a2c7db83b430ba-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/A 2023-04-16 00:53:44,070 INFO [StoreOpener-c9039369968c422a52a2c7db83b430ba-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-16 00:53:44,071 INFO [StoreOpener-c9039369968c422a52a2c7db83b430ba-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region c9039369968c422a52a2c7db83b430ba columnFamilyName A 2023-04-16 00:53:44,071 INFO [StoreOpener-c9039369968c422a52a2c7db83b430ba-1] regionserver.HStore(310): Store=c9039369968c422a52a2c7db83b430ba/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-16 00:53:44,072 INFO [StoreOpener-c9039369968c422a52a2c7db83b430ba-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region c9039369968c422a52a2c7db83b430ba 2023-04-16 00:53:44,073 DEBUG [StoreOpener-c9039369968c422a52a2c7db83b430ba-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/B 2023-04-16 00:53:44,073 DEBUG [StoreOpener-c9039369968c422a52a2c7db83b430ba-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/B 2023-04-16 00:53:44,074 INFO [StoreOpener-c9039369968c422a52a2c7db83b430ba-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-16 00:53:44,074 INFO [StoreOpener-c9039369968c422a52a2c7db83b430ba-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region c9039369968c422a52a2c7db83b430ba columnFamilyName B 2023-04-16 00:53:44,075 INFO [StoreOpener-c9039369968c422a52a2c7db83b430ba-1] regionserver.HStore(310): Store=c9039369968c422a52a2c7db83b430ba/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-16 00:53:44,075 INFO [StoreOpener-c9039369968c422a52a2c7db83b430ba-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region c9039369968c422a52a2c7db83b430ba 2023-04-16 00:53:44,076 DEBUG [StoreOpener-c9039369968c422a52a2c7db83b430ba-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/C 2023-04-16 00:53:44,077 DEBUG [StoreOpener-c9039369968c422a52a2c7db83b430ba-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/C 2023-04-16 00:53:44,077 INFO [StoreOpener-c9039369968c422a52a2c7db83b430ba-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-16 00:53:44,077 INFO [StoreOpener-c9039369968c422a52a2c7db83b430ba-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region c9039369968c422a52a2c7db83b430ba columnFamilyName C 2023-04-16 00:53:44,078 INFO [StoreOpener-c9039369968c422a52a2c7db83b430ba-1] regionserver.HStore(310): Store=c9039369968c422a52a2c7db83b430ba/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-16 00:53:44,079 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. 2023-04-16 00:53:44,081 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba 2023-04-16 00:53:44,081 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba 2023-04-16 00:53:44,084 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-16 00:53:44,086 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for c9039369968c422a52a2c7db83b430ba 2023-04-16 00:53:44,089 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-16 00:53:44,090 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened c9039369968c422a52a2c7db83b430ba; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=67385912, jitterRate=0.0041283369064331055}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-16 00:53:44,090 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for c9039369968c422a52a2c7db83b430ba: 2023-04-16 00:53:44,091 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba., pid=11, masterSystemTime=1681606424057 2023-04-16 00:53:44,094 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. 2023-04-16 00:53:44,094 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. 2023-04-16 00:53:44,095 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=c9039369968c422a52a2c7db83b430ba, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:44,095 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681606424095"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1681606424095"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1681606424095"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1681606424095"}]},"ts":"1681606424095"} 2023-04-16 00:53:44,101 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=11, resume processing ppid=10 2023-04-16 00:53:44,101 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=11, ppid=10, state=SUCCESS; OpenRegionProcedure c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 in 195 msec 2023-04-16 00:53:44,105 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=10, resume processing ppid=9 2023-04-16 00:53:44,105 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=10, ppid=9, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=c9039369968c422a52a2c7db83b430ba, ASSIGN in 360 msec 2023-04-16 00:53:44,106 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-16 00:53:44,106 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681606424106"}]},"ts":"1681606424106"} 2023-04-16 00:53:44,109 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-16 00:53:44,112 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-16 00:53:44,115 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=9, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 443 msec 2023-04-16 00:53:44,299 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-04-16 00:53:44,300 INFO [Listener at localhost/33477] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 9 completed 2023-04-16 00:53:44,305 DEBUG [Listener at localhost/33477] zookeeper.ReadOnlyZKClient(139): Connect 0x2678d495 to 127.0.0.1:53150 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-16 00:53:44,310 DEBUG [Listener at localhost/33477] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@8c3f619, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-16 00:53:44,312 DEBUG [Listener at localhost/33477] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-16 00:53:44,316 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:39978, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-16 00:53:44,319 DEBUG [Listener at localhost/33477] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-16 00:53:44,321 INFO [RS-EventLoopGroup-1-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:35168, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-16 00:53:44,330 DEBUG [Listener at localhost/33477] zookeeper.ReadOnlyZKClient(139): Connect 0x1b886c8f to 127.0.0.1:53150 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-16 00:53:44,336 DEBUG [Listener at localhost/33477] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@748f869b, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-16 00:53:44,336 DEBUG [Listener at localhost/33477] zookeeper.ReadOnlyZKClient(139): Connect 0x0e16d41b to 127.0.0.1:53150 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-16 00:53:44,342 DEBUG [Listener at localhost/33477] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5e91d243, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-16 00:53:44,343 DEBUG [Listener at localhost/33477] zookeeper.ReadOnlyZKClient(139): Connect 0x77648431 to 127.0.0.1:53150 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-16 00:53:44,347 DEBUG [Listener at localhost/33477] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@30eb7fb2, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-16 00:53:44,347 DEBUG [Listener at localhost/33477] zookeeper.ReadOnlyZKClient(139): Connect 0x7e728705 to 127.0.0.1:53150 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-16 00:53:44,353 DEBUG [Listener at localhost/33477] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@46192d17, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-16 00:53:44,354 DEBUG [Listener at localhost/33477] zookeeper.ReadOnlyZKClient(139): Connect 0x09a575b8 to 127.0.0.1:53150 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-16 00:53:44,362 DEBUG [Listener at localhost/33477] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2c57e1b, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-16 00:53:44,363 DEBUG [Listener at localhost/33477] zookeeper.ReadOnlyZKClient(139): Connect 0x1818ad0a to 127.0.0.1:53150 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-16 00:53:44,372 DEBUG [Listener at localhost/33477] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@49d71120, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-16 00:53:44,372 DEBUG [Listener at localhost/33477] zookeeper.ReadOnlyZKClient(139): Connect 0x04fb2b82 to 127.0.0.1:53150 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-16 00:53:44,412 DEBUG [Listener at localhost/33477] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@59d6b6ea, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-16 00:53:44,413 DEBUG [Listener at localhost/33477] zookeeper.ReadOnlyZKClient(139): Connect 0x6129209e to 127.0.0.1:53150 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-16 00:53:44,418 DEBUG [Listener at localhost/33477] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@d24bbb7, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-16 00:53:44,418 DEBUG [Listener at localhost/33477] zookeeper.ReadOnlyZKClient(139): Connect 0x4b02eeb3 to 127.0.0.1:53150 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-16 00:53:44,422 DEBUG [Listener at localhost/33477] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@63871a9d, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-16 00:53:44,435 DEBUG [hconnection-0x3ea07dd7-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-16 00:53:44,436 DEBUG [hconnection-0x2df665ba-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-16 00:53:44,440 DEBUG [hconnection-0x67c33566-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-16 00:53:44,440 DEBUG [hconnection-0x2770c261-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-16 00:53:44,440 DEBUG [hconnection-0x12799251-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-16 00:53:44,440 DEBUG [hconnection-0x5343c07a-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-16 00:53:44,443 DEBUG [hconnection-0x2a9695a6-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-16 00:53:44,445 DEBUG [hconnection-0x6ccd5026-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-16 00:53:44,446 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:39990, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-16 00:53:44,453 DEBUG [hconnection-0x5d79b519-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-16 00:53:44,455 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:40000, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-16 00:53:44,456 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:39980, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-16 00:53:44,457 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:40014, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-16 00:53:44,459 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:40020, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-16 00:53:44,465 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:40044, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-16 00:53:44,476 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:40046, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-16 00:53:44,476 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:40052, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-16 00:53:44,478 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:40032, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-16 00:53:44,481 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:53:44,506 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:53:44,520 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:53:44,520 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:53:44,521 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:53:44,521 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:53:44,522 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:44,522 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:44,523 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:53:44,523 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:53:44,523 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:44,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:53:44,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:44,525 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:44,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:53:44,526 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:44,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:44,526 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:53:44,526 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:53:44,544 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:53:44,546 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(9158): Flush requested on c9039369968c422a52a2c7db83b430ba 2023-04-16 00:53:44,546 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:53:44,546 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:53:44,551 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing c9039369968c422a52a2c7db83b430ba 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-16 00:53:44,552 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:53:44,594 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:53:44,595 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:53:44,598 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. 2023-04-16 00:53:44,599 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. started... 2023-04-16 00:53:44,599 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. as already flushing 2023-04-16 00:53:44,601 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. 2023-04-16 00:53:44,606 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:53:44,606 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:53:44,606 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:44,606 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:53:44,606 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:53:44,617 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c9039369968c422a52a2c7db83b430ba, store=A 2023-04-16 00:53:44,618 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:53:44,619 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c9039369968c422a52a2c7db83b430ba, store=B 2023-04-16 00:53:44,619 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:53:44,619 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c9039369968c422a52a2c7db83b430ba, store=C 2023-04-16 00:53:44,619 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:53:44,627 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:44,627 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:44,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:44,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:53:44,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:53:44,627 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:44,628 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:53:44,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:53:44,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:53:44,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:44,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:44,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:53:44,633 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:53:44,639 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:53:44,639 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2c8bfbe2[Count = 0] remaining members to acquire global barrier 2023-04-16 00:53:44,639 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:44,641 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:44,642 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:44,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:44,642 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:53:44,642 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:53:44,642 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:53:44,642 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:44,642 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:53:44,646 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:44,646 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:53:44,646 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:53:44,646 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:53:44,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:44,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:53:44,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:53:44,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:53:44,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:53:44,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:44,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:44,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:53:44,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:44,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:44,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:53:44,653 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:53:44,653 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:53:44,653 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:53:44,653 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:53:44,653 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:53:44,653 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:53:44,665 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:44,665 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:53:44,665 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:44,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:44,665 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:44,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:44,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:53:44,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:53:44,666 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:53:44,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:53:44,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:44,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:53:44,667 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:44,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:44,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:53:44,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:44,670 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:44,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:44,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:53:44,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:44,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:44,690 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:53:44,690 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:44,690 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:53:44,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:53:44,690 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:44,691 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:53:44,691 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:53:44,690 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:53:44,691 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:53:44,691 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:44,691 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:44,691 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:44,691 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:44,691 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:53:44,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:53:44,691 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:44,697 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:53:44,698 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:53:44,698 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:53:44,743 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681606424684, max timestamp=1681606424724Num uniques 859; 2023-04-16 00:53:44,743 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-16 00:53:44,758 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681606424703, max timestamp=1681606424724Num uniques 859; 2023-04-16 00:53:44,759 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-16 00:53:44,761 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681606424703, max timestamp=1681606424724Num uniques 859; 2023-04-16 00:53:44,761 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-16 00:53:44,798 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:53:44,805 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:53:44,814 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:53:44,821 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:53:44,822 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:53:44,822 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:53:44,822 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:53:44,823 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:53:44,823 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:53:44,824 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:44,824 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:44,825 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:44,825 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:53:44,826 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:53:44,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:53:44,826 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:44,826 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:53:44,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:44,827 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:44,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:53:44,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:44,828 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:53:44,828 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:53:44,828 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:53:44,828 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:53:44,828 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:53:44,829 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:53:44,829 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:53:44,829 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. 2023-04-16 00:53:44,829 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:53:44,829 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. started... 2023-04-16 00:53:44,830 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. as already flushing 2023-04-16 00:53:44,830 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. 2023-04-16 00:53:44,831 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:53:44,831 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:53:44,831 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:44,831 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:53:44,831 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:53:44,834 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:44,834 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:44,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:44,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:53:44,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:53:44,835 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:44,835 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:53:44,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:53:44,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:53:44,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:44,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:44,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:53:44,838 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:53:44,838 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@476ab76b[Count = 0] remaining members to acquire global barrier 2023-04-16 00:53:44,838 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:53:44,838 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:44,839 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:44,840 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:44,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:44,840 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:53:44,840 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:53:44,840 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:53:44,840 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:44,840 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:53:44,842 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:53:44,842 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:44,843 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:53:44,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:44,843 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:53:44,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:53:44,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:53:44,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:53:44,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:53:44,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:44,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:44,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:53:44,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:44,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:44,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:53:44,860 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:53:44,861 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:53:44,861 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:53:44,861 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:53:44,861 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:53:44,861 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:53:44,867 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:53:44,868 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40046 deadline: 1681606484854, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:44,868 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:44,869 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:53:44,869 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:44,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:44,869 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:44,869 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:53:44,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:53:44,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:44,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:53:44,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:53:44,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:44,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:53:44,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:44,871 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:44,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:53:44,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:44,873 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:53:44,874 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40000 deadline: 1681606484858, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:44,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:44,875 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:53:44,875 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39990 deadline: 1681606484861, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:44,876 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:53:44,876 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40044 deadline: 1681606484867, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:44,877 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:53:44,878 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39980 deadline: 1681606484868, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:44,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:53:44,882 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:44,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:44,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:44,895 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:44,895 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:53:44,895 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:44,895 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:53:44,895 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:44,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:53:44,895 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:53:44,896 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:53:44,895 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:53:44,896 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:53:44,897 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:53:44,895 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:53:44,895 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:44,897 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:53:44,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:53:44,897 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:44,897 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:44,897 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:44,997 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:53:44,998 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:53:45,005 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:53:45,014 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:53:45,014 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40046 deadline: 1681606485013, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:45,015 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:53:45,015 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39990 deadline: 1681606485014, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:45,016 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:53:45,017 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40000 deadline: 1681606485016, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:45,021 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:53:45,022 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39980 deadline: 1681606485019, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:45,023 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:53:45,023 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40044 deadline: 1681606485019, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:45,036 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:53:45,036 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:53:45,036 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:53:45,036 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:53:45,044 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:53:45,044 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:53:45,045 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:45,045 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:45,046 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:45,048 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:45,048 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:53:45,050 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:53:45,050 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:53:45,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:53:45,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:45,052 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:45,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:53:45,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:45,053 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:53:45,055 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:53:45,055 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:53:45,055 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:53:45,055 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:53:45,056 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:53:45,056 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:53:45,056 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:53:45,056 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. 2023-04-16 00:53:45,056 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. started... 2023-04-16 00:53:45,057 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. as already flushing 2023-04-16 00:53:45,057 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. 2023-04-16 00:53:45,059 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:53:45,059 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:53:45,059 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:45,060 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:53:45,060 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:53:45,062 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:45,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:45,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:53:45,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:53:45,067 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:45,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:53:45,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:53:45,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:45,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:45,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:53:45,071 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:53:45,071 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@69c668d5[Count = 0] remaining members to acquire global barrier 2023-04-16 00:53:45,071 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:53:45,071 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:45,072 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:45,072 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:53:45,074 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:45,077 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:45,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:45,077 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:53:45,077 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:53:45,077 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:53:45,079 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:45,079 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:53:45,081 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:53:45,081 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:53:45,081 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:53:45,083 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:45,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:45,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:53:45,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:53:45,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:53:45,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:53:45,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:45,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:45,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:53:45,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:45,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:45,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:53:45,089 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:53:45,089 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:53:45,089 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:53:45,089 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:53:45,089 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:53:45,089 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:53:45,092 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:45,092 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:53:45,092 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:45,092 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:45,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:45,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:53:45,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:45,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:53:45,093 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:53:45,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:53:45,094 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:45,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:53:45,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:45,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:45,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:53:45,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:45,096 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:45,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:45,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:53:45,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:45,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:45,107 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:53:45,107 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:53:45,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:53:45,107 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:45,107 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:45,107 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:53:45,107 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:53:45,107 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:53:45,108 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:53:45,107 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:45,108 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:45,108 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:45,109 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:45,109 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:53:45,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:53:45,109 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:45,109 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:53:45,109 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:53:45,109 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:53:45,201 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=12 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/.tmp/A/5060fe90e1494ff1977278cbc188497d 2023-04-16 00:53:45,209 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:53:45,211 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:53:45,219 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:53:45,219 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40046 deadline: 1681606485217, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:45,221 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:53:45,222 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:53:45,222 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39990 deadline: 1681606485219, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:45,223 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:53:45,223 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40000 deadline: 1681606485221, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:45,227 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:53:45,227 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39980 deadline: 1681606485226, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:45,229 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:53:45,230 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40044 deadline: 1681606485228, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:45,232 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:53:45,232 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:53:45,232 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:53:45,232 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:53:45,234 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:53:45,234 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:53:45,235 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:45,235 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:45,238 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:45,238 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:53:45,238 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:53:45,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:53:45,239 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:45,239 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:53:45,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:45,240 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:45,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:53:45,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:45,241 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:53:45,242 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:53:45,245 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:53:45,245 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:53:45,245 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:53:45,247 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:53:45,247 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:53:45,247 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:53:45,247 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. 2023-04-16 00:53:45,247 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. started... 2023-04-16 00:53:45,247 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. as already flushing 2023-04-16 00:53:45,247 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. 2023-04-16 00:53:45,248 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:53:45,248 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:53:45,248 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:45,248 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:53:45,248 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:53:45,251 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:45,251 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:45,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:45,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:53:45,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:53:45,251 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:45,251 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:53:45,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:53:45,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:53:45,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:45,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:45,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:53:45,254 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:53:45,254 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2979d248[Count = 0] remaining members to acquire global barrier 2023-04-16 00:53:45,254 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:53:45,254 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:45,256 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:45,256 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:53:45,264 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:45,264 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:45,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:45,264 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:53:45,264 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:53:45,264 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:53:45,273 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:53:45,273 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:45,273 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:53:45,273 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:53:45,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:45,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:53:45,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:53:45,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:53:45,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:53:45,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:45,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:45,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:53:45,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:45,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:45,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:53:45,280 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:53:45,280 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:53:45,280 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:53:45,280 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:53:45,280 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:53:45,280 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:53:45,286 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:45,286 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:45,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:45,286 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:45,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:53:45,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:53:45,286 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:53:45,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:45,287 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:53:45,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:53:45,287 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:45,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:53:45,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:45,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:45,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:53:45,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:45,290 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:45,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:45,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:53:45,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:45,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:45,298 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:53:45,298 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:53:45,298 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:53:45,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:53:45,299 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:45,299 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:45,299 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:45,299 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:45,299 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:45,299 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:53:45,299 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:45,299 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:45,299 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:53:45,300 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:53:45,300 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:53:45,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:53:45,300 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:53:45,300 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:53:45,303 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:53:45,362 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=12 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/.tmp/B/bf4078543a914d21bb8934742f1d599a 2023-04-16 00:53:45,401 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:53:45,402 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:53:45,406 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:53:45,432 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:53:45,432 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:53:45,432 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:53:45,433 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:53:45,433 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:53:45,433 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:53:45,436 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:45,437 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:45,438 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:53:45,438 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:53:45,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:53:45,439 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:45,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:45,440 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:45,440 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:45,440 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:53:45,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:53:45,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:45,442 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:53:45,442 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:53:45,442 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:53:45,442 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:53:45,442 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:53:45,443 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:53:45,443 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:53:45,443 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:53:45,443 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. 2023-04-16 00:53:45,444 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. started... 2023-04-16 00:53:45,444 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. as already flushing 2023-04-16 00:53:45,444 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. 2023-04-16 00:53:45,446 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:53:45,446 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:53:45,446 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:45,447 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:53:45,447 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:53:45,451 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:45,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:45,451 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:45,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:53:45,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:53:45,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:53:45,452 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:45,452 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:53:45,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:53:45,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:45,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:45,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:53:45,455 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:53:45,455 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5a5e686e[Count = 0] remaining members to acquire global barrier 2023-04-16 00:53:45,455 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:53:45,455 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:45,458 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:45,458 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:45,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:45,458 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:53:45,458 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:53:45,458 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:53:45,458 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:45,459 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:53:45,460 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:45,460 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:53:45,460 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:53:45,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:45,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:53:45,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:53:45,460 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:53:45,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:53:45,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:53:45,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:45,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:45,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:53:45,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:45,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:45,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:53:45,470 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:53:45,470 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:53:45,470 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:53:45,470 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:53:45,470 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:53:45,470 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:53:45,474 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:45,474 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:45,474 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:45,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:45,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:53:45,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:53:45,474 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:53:45,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:45,475 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:53:45,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:53:45,475 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:45,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:53:45,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:45,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:45,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:53:45,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:45,478 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:45,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:45,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:53:45,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:45,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:45,486 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:53:45,486 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:53:45,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:53:45,486 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:53:45,486 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:45,486 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:45,486 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:45,486 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:45,486 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:45,486 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:45,486 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:45,486 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:53:45,486 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:53:45,486 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:53:45,487 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:53:45,487 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:53:45,487 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:53:45,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:53:45,487 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:53:45,532 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:53:45,533 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40046 deadline: 1681606485523, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:45,534 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:53:45,534 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39990 deadline: 1681606485531, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:45,534 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:53:45,535 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39980 deadline: 1681606485532, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:45,535 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:53:45,535 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40000 deadline: 1681606485532, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:45,537 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:53:45,556 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40044 deadline: 1681606485534, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:45,587 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:53:45,589 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:53:45,596 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:53:45,612 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:53:45,612 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:53:45,612 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:53:45,612 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:53:45,617 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:53:45,617 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:53:45,617 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:45,617 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:45,619 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:53:45,620 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:53:45,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:53:45,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:45,621 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:45,621 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:45,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:53:45,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:45,621 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:45,621 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:53:45,621 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:53:45,640 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:53:45,640 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:53:45,640 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:53:45,640 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:53:45,649 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:53:45,651 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:53:45,651 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:53:45,656 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. 2023-04-16 00:53:45,656 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. started... 2023-04-16 00:53:45,656 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. as already flushing 2023-04-16 00:53:45,656 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. 2023-04-16 00:53:45,659 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:53:45,659 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:53:45,659 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:45,659 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:53:45,659 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:53:45,661 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:45,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:45,662 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:45,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:53:45,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:53:45,662 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:45,662 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:53:45,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:53:45,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:53:45,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:45,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:45,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:53:45,666 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:53:45,666 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@481d5da1[Count = 0] remaining members to acquire global barrier 2023-04-16 00:53:45,667 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:53:45,667 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:45,668 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:45,668 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:45,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:45,668 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:53:45,668 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:53:45,668 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:53:45,668 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:45,669 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:53:45,671 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:53:45,671 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:45,671 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:53:45,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:45,671 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:53:45,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:53:45,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:53:45,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:53:45,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:53:45,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:45,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:45,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:53:45,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:45,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:45,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:53:45,685 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:53:45,685 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:53:45,685 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:53:45,685 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:53:45,685 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:53:45,685 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:53:45,690 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:45,691 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:53:45,691 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:45,691 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:45,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:45,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:45,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:53:45,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:53:45,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:53:45,692 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:45,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:45,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:53:45,695 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:53:45,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:53:45,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:45,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:45,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:45,696 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:45,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:53:45,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:45,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:45,700 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:53:45,700 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:53:45,700 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:53:45,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:53:45,701 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:53:45,701 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:53:45,701 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:53:45,701 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:53:45,701 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:45,701 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:45,701 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:45,701 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:53:45,701 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:53:45,701 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:45,701 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:45,701 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:53:45,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:53:45,701 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:45,702 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:45,801 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:53:45,802 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:53:45,806 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:53:45,809 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:53:45,809 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:53:45,809 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:53:45,809 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:53:45,810 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:53:45,810 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:53:45,811 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:45,811 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:45,813 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:45,813 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:53:45,813 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:53:45,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:53:45,813 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:45,813 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:53:45,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:45,814 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:45,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:53:45,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:45,815 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:53:45,815 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:53:45,815 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:53:45,815 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:53:45,815 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:53:45,816 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:53:45,817 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:53:45,817 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:53:45,817 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. 2023-04-16 00:53:45,817 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. started... 2023-04-16 00:53:45,817 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. as already flushing 2023-04-16 00:53:45,817 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. 2023-04-16 00:53:45,818 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:53:45,818 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:53:45,818 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:45,819 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:53:45,819 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:53:45,825 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:45,825 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:45,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:45,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:53:45,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:53:45,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:53:45,826 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:45,826 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:53:45,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:53:45,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:45,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:45,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:53:45,828 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:53:45,829 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@12c704b7[Count = 0] remaining members to acquire global barrier 2023-04-16 00:53:45,829 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:53:45,829 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:45,830 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:45,830 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:45,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:45,830 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:53:45,830 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:53:45,830 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:53:45,830 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:45,830 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:53:45,832 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:53:45,832 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:53:45,832 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:53:45,832 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:45,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:45,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:53:45,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:53:45,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:53:45,837 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=12 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/.tmp/C/0da3dfc269354e279ba38e2eed98439d 2023-04-16 00:53:45,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:53:45,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:45,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:45,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:53:45,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:45,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:45,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:53:45,853 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:53:45,853 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:53:45,853 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:53:45,853 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:53:45,853 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:53:45,853 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:53:45,855 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/.tmp/A/5060fe90e1494ff1977278cbc188497d as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/A/5060fe90e1494ff1977278cbc188497d 2023-04-16 00:53:45,869 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:45,869 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:53:45,869 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:45,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:45,870 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:45,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:45,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:53:45,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:53:45,870 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:53:45,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:53:45,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:45,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:53:45,871 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:45,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:45,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:53:45,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:45,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:45,876 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:45,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:53:45,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:45,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:45,880 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/A/5060fe90e1494ff1977278cbc188497d, entries=150, sequenceid=12, filesize=11.6 K 2023-04-16 00:53:45,884 DEBUG [MemStoreFlusher.0] regionserver.StoreScanner(961): StoreScanner already closing. There is no need to updateReaders 2023-04-16 00:53:45,886 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:45,886 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:45,886 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:53:45,886 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:45,886 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:53:45,886 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:53:45,886 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:53:45,886 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:53:45,887 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:53:45,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:53:45,886 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:45,888 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:45,888 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:45,888 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:45,888 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:53:45,888 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:53:45,888 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:53:45,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:53:45,889 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/.tmp/B/bf4078543a914d21bb8934742f1d599a as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/B/bf4078543a914d21bb8934742f1d599a 2023-04-16 00:53:45,907 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/B/bf4078543a914d21bb8934742f1d599a, entries=150, sequenceid=12, filesize=11.6 K 2023-04-16 00:53:45,909 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/.tmp/C/0da3dfc269354e279ba38e2eed98439d as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/C/0da3dfc269354e279ba38e2eed98439d 2023-04-16 00:53:45,924 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/C/0da3dfc269354e279ba38e2eed98439d, entries=150, sequenceid=12, filesize=11.6 K 2023-04-16 00:53:45,927 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=125.23 KB/128240 for c9039369968c422a52a2c7db83b430ba in 1381ms, sequenceid=12, compaction requested=false 2023-04-16 00:53:45,929 DEBUG [MemStoreFlusher.0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'TestAcidGuarantees' 2023-04-16 00:53:45,932 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for c9039369968c422a52a2c7db83b430ba: 2023-04-16 00:53:45,988 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:53:45,989 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:53:45,994 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:53:45,996 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:53:45,997 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:53:45,997 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:53:45,997 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:53:46,001 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:53:46,001 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:53:46,002 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:46,002 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:46,003 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:46,003 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:53:46,004 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:53:46,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:53:46,004 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:46,004 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:53:46,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:46,005 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:46,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:53:46,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:46,005 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:53:46,006 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:53:46,006 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:53:46,006 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:53:46,006 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:53:46,009 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:53:46,020 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:53:46,020 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. 2023-04-16 00:53:46,020 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:53:46,020 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. started... 2023-04-16 00:53:46,020 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing c9039369968c422a52a2c7db83b430ba 3/3 column families, dataSize=125.23 KB heapSize=355.30 KB 2023-04-16 00:53:46,020 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c9039369968c422a52a2c7db83b430ba, store=A 2023-04-16 00:53:46,021 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 00:53:46,021 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c9039369968c422a52a2c7db83b430ba, store=B 2023-04-16 00:53:46,021 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 00:53:46,021 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c9039369968c422a52a2c7db83b430ba, store=C 2023-04-16 00:53:46,021 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 00:53:46,056 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. as already flushing 2023-04-16 00:53:46,056 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(9158): Flush requested on c9039369968c422a52a2c7db83b430ba 2023-04-16 00:53:46,089 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:53:46,089 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40000 deadline: 1681606486068, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:46,092 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:53:46,092 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40046 deadline: 1681606486071, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:46,100 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=47 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/.tmp/A/b395a4b8fa144ace900151c416009e87 2023-04-16 00:53:46,107 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:53:46,107 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39980 deadline: 1681606486076, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:46,108 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:53:46,108 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40044 deadline: 1681606486084, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:46,109 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:53:46,109 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39990 deadline: 1681606486087, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:46,157 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=47 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/.tmp/B/430794e955ce43c284188aef09954da9 2023-04-16 00:53:46,215 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=47 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/.tmp/C/cca93818a0814e2b8ad85e1628228b7b 2023-04-16 00:53:46,222 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:53:46,222 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40046 deadline: 1681606486197, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:46,223 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:53:46,223 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40000 deadline: 1681606486197, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:46,228 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/.tmp/A/b395a4b8fa144ace900151c416009e87 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/A/b395a4b8fa144ace900151c416009e87 2023-04-16 00:53:46,238 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/A/b395a4b8fa144ace900151c416009e87, entries=150, sequenceid=47, filesize=11.6 K 2023-04-16 00:53:46,241 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:53:46,245 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39980 deadline: 1681606486211, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:46,245 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:53:46,246 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40044 deadline: 1681606486215, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:46,246 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:53:46,246 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39990 deadline: 1681606486217, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:46,250 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/.tmp/B/430794e955ce43c284188aef09954da9 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/B/430794e955ce43c284188aef09954da9 2023-04-16 00:53:46,262 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/B/430794e955ce43c284188aef09954da9, entries=150, sequenceid=47, filesize=11.6 K 2023-04-16 00:53:46,272 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/.tmp/C/cca93818a0814e2b8ad85e1628228b7b as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/C/cca93818a0814e2b8ad85e1628228b7b 2023-04-16 00:53:46,292 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/C/cca93818a0814e2b8ad85e1628228b7b, entries=150, sequenceid=47, filesize=11.6 K 2023-04-16 00:53:46,295 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~125.23 KB/128240, heapSize ~355.25 KB/363776, currentSize=53.67 KB/54960 for c9039369968c422a52a2c7db83b430ba in 274ms, sequenceid=47, compaction requested=false 2023-04-16 00:53:46,296 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for c9039369968c422a52a2c7db83b430ba: 2023-04-16 00:53:46,296 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. 2023-04-16 00:53:46,296 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:53:46,297 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:53:46,297 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:46,297 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:53:46,297 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:53:46,300 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:46,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:46,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:53:46,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:53:46,301 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:46,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:53:46,301 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:46,301 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:53:46,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:53:46,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:46,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:46,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:53:46,305 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:53:46,305 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@75cb6f35[Count = 0] remaining members to acquire global barrier 2023-04-16 00:53:46,305 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:53:46,305 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:46,306 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:46,306 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:46,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:46,306 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:53:46,306 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:53:46,306 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:53:46,307 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:46,307 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:53:46,310 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:46,310 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:53:46,310 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:53:46,310 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:53:46,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:46,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:53:46,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:53:46,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:53:46,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:53:46,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:46,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:46,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:53:46,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:46,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:46,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:53:46,318 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:53:46,318 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:53:46,318 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:53:46,318 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:53:46,318 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:53:46,318 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:53:46,320 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:46,320 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:46,320 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:46,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:46,320 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:53:46,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:46,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:53:46,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:53:46,321 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:53:46,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:53:46,321 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:46,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:53:46,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:46,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:46,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:53:46,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:46,323 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:46,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:46,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:53:46,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:46,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:46,334 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:53:46,334 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:53:46,334 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:53:46,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:53:46,334 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:53:46,334 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:53:46,340 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:53:46,334 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:46,340 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:46,340 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:46,340 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:46,340 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:46,340 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:46,340 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:46,341 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:53:46,341 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:53:46,347 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:53:46,347 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:53:46,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:53:46,439 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing c9039369968c422a52a2c7db83b430ba 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-16 00:53:46,440 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(9158): Flush requested on c9039369968c422a52a2c7db83b430ba 2023-04-16 00:53:46,440 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c9039369968c422a52a2c7db83b430ba, store=A 2023-04-16 00:53:46,440 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:53:46,440 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c9039369968c422a52a2c7db83b430ba, store=B 2023-04-16 00:53:46,440 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:53:46,440 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c9039369968c422a52a2c7db83b430ba, store=C 2023-04-16 00:53:46,440 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:53:46,441 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:53:46,442 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:53:46,446 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:53:46,465 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:meta' 2023-04-16 00:53:46,468 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:namespace' 2023-04-16 00:53:46,475 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:53:46,480 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:53:46,480 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:53:46,485 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:53:46,492 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:53:46,492 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:53:46,493 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:46,493 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:46,496 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:53:46,496 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:46,496 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:53:46,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:53:46,496 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:46,496 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:53:46,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:46,504 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:46,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:53:46,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:46,508 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:53:46,520 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:53:46,520 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:53:46,527 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:53:46,527 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:53:46,527 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=59 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/.tmp/A/04df6097d94a4008940de9cc7c096e60 2023-04-16 00:53:46,537 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:53:46,548 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:53:46,548 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. 2023-04-16 00:53:46,548 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. started... 2023-04-16 00:53:46,548 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:53:46,557 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. as already flushing 2023-04-16 00:53:46,557 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. 2023-04-16 00:53:46,563 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:53:46,563 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:53:46,563 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:46,563 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:53:46,563 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:53:46,565 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:46,565 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:46,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:46,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:53:46,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:53:46,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:53:46,566 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:46,566 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:53:46,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:53:46,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:46,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:46,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:53:46,568 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:53:46,568 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5c7f64fa[Count = 0] remaining members to acquire global barrier 2023-04-16 00:53:46,568 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:53:46,568 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:46,570 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:46,570 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:46,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:46,570 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:53:46,570 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:53:46,570 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:53:46,571 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:46,571 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:53:46,572 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:53:46,572 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:53:46,572 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:53:46,572 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:46,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:46,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:53:46,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:53:46,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:53:46,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:53:46,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:46,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:46,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:53:46,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:46,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:46,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:53:46,586 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:53:46,586 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:53:46,586 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:53:46,586 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:53:46,586 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:53:46,586 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:53:46,589 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:46,589 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:53:46,589 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:46,589 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:46,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:46,589 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:53:46,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:46,590 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:46,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:53:46,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:53:46,590 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40044 deadline: 1681606486571, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:46,590 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:53:46,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:53:46,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:46,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:53:46,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:46,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:53:46,594 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:46,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:46,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:46,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:53:46,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:46,598 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:53:46,598 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:53:46,598 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:46,598 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:46,598 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:53:46,598 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:53:46,598 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:53:46,598 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:53:46,598 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:46,598 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:46,598 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:46,598 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:53:46,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:53:46,599 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:53:46,598 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:46,599 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:46,599 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:53:46,599 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:53:46,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:53:46,614 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1681606426563, max timestamp=1681606426584Num uniques 859; 2023-04-16 00:53:46,614 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-16 00:53:46,616 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681606426563, max timestamp=1681606426584Num uniques 859; 2023-04-16 00:53:46,617 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-16 00:53:46,634 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1681606426563, max timestamp=1681606426584Num uniques 859; 2023-04-16 00:53:46,634 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=6870, suffix heap size=117064,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=150 2023-04-16 00:53:46,699 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:53:46,700 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:53:46,704 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:53:46,706 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:53:46,706 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:53:46,706 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:53:46,706 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:53:46,712 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:53:46,712 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:53:46,713 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:46,713 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:46,715 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:46,715 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:53:46,715 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:53:46,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:53:46,716 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:46,716 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:53:46,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:46,724 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:46,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:53:46,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:46,730 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:53:46,730 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:53:46,730 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:53:46,731 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:53:46,731 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:53:46,733 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:53:46,733 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:53:46,734 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:53:46,734 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. 2023-04-16 00:53:46,734 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. started... 2023-04-16 00:53:46,734 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. as already flushing 2023-04-16 00:53:46,734 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. 2023-04-16 00:53:46,738 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:53:46,738 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39990 deadline: 1681606486701, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:46,739 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:53:46,739 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:53:46,739 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:46,739 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:53:46,739 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:53:46,740 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:53:46,742 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40000 deadline: 1681606486716, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:46,743 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:46,743 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:46,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:46,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:53:46,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:53:46,744 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:46,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:53:46,744 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:53:46,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:53:46,744 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:53:46,745 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39980 deadline: 1681606486736, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:46,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:46,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:46,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:53:46,746 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:53:46,746 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7eadda81[Count = 0] remaining members to acquire global barrier 2023-04-16 00:53:46,746 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:53:46,746 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:46,747 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:53:46,747 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40044 deadline: 1681606486738, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:46,748 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:53:46,748 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40046 deadline: 1681606486739, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:46,749 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:46,749 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:46,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:46,749 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:53:46,749 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:53:46,749 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:53:46,749 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:46,750 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:53:46,751 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:46,751 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:53:46,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:46,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:53:46,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:53:46,751 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:53:46,751 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:53:46,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:53:46,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:53:46,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:46,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:46,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:53:46,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:46,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:46,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:53:46,754 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:53:46,754 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:53:46,754 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:53:46,754 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:53:46,754 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:53:46,754 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:53:46,755 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:46,756 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:46,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:46,756 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:46,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:53:46,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:53:46,756 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:53:46,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:46,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:53:46,767 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:53:46,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:53:46,767 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:46,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:46,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:46,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:53:46,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:46,772 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:46,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:46,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:53:46,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:46,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:46,776 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:53:46,776 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:53:46,776 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:53:46,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:53:46,776 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:53:46,776 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:46,776 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:53:46,776 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:53:46,776 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:46,776 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:46,777 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:46,777 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:46,777 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:46,777 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:46,777 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:53:46,777 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:53:46,778 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:53:46,778 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:53:46,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:53:46,851 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:53:46,852 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39990 deadline: 1681606486844, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:46,852 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:53:46,852 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40000 deadline: 1681606486844, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:46,854 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:53:46,855 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39980 deadline: 1681606486848, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:46,856 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:53:46,858 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40044 deadline: 1681606486850, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:46,859 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:53:46,859 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40046 deadline: 1681606486850, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:46,877 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:53:46,878 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:53:46,882 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:53:46,894 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:53:46,894 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:53:46,894 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:53:46,894 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:53:46,895 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:53:46,895 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:53:46,897 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:46,897 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:46,899 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:53:46,899 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:46,899 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:53:46,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:53:46,900 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:46,900 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:53:46,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:46,900 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:46,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:53:46,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:46,901 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:53:46,911 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:53:46,911 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:53:46,911 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:53:46,911 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:53:46,913 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:53:46,920 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:53:46,920 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:53:46,920 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. 2023-04-16 00:53:46,920 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. started... 2023-04-16 00:53:46,920 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. as already flushing 2023-04-16 00:53:46,920 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. 2023-04-16 00:53:46,923 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:53:46,923 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:53:46,923 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:46,923 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:53:46,923 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:53:46,926 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:46,926 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:46,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:46,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:53:46,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:53:46,927 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:46,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:53:46,927 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:53:46,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:53:46,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:46,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:46,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:53:46,941 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:53:46,941 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@54e59b5f[Count = 0] remaining members to acquire global barrier 2023-04-16 00:53:46,941 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:53:46,941 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:46,943 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:46,943 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:46,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:46,943 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:53:46,943 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:53:46,943 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:53:46,944 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:46,944 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:53:46,945 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:53:46,945 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:53:46,945 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:53:46,945 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:46,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:46,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:53:46,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:53:46,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:53:46,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:53:46,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:46,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:46,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:53:46,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:46,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:46,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:53:46,962 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:53:46,962 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:53:46,962 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:53:46,962 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:53:46,962 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:53:46,963 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:53:46,966 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:46,966 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:46,966 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:46,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:46,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:53:46,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:53:46,966 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:53:46,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:46,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:53:46,967 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:46,967 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:53:46,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:53:46,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:46,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:46,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:53:46,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:46,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:46,969 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:46,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:53:46,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:46,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:46,973 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:53:46,985 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:46,985 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:46,985 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:46,985 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:46,985 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:46,985 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:46,985 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:46,986 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:53:46,986 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:53:46,986 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:53:46,986 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:53:46,986 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:53:46,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:53:46,986 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:53:46,986 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:53:46,987 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:53:46,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:53:47,049 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=59 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/.tmp/B/6c92c274cda945e9adc21f14b3b3c391 2023-04-16 00:53:47,069 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:53:47,069 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40044 deadline: 1681606487063, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:47,082 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:53:47,083 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39980 deadline: 1681606487069, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:47,083 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:53:47,083 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40000 deadline: 1681606487071, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:47,084 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:53:47,084 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40046 deadline: 1681606487071, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:47,086 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:53:47,088 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:53:47,090 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:53:47,090 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39990 deadline: 1681606487085, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:47,091 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:53:47,101 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:53:47,101 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:53:47,101 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:53:47,103 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:53:47,104 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:53:47,104 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:53:47,105 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:47,105 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:47,106 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:47,106 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:53:47,107 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:53:47,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:53:47,107 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:47,107 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:53:47,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:47,108 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:47,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:53:47,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:47,109 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:53:47,120 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:53:47,121 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:53:47,121 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:53:47,121 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:53:47,137 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:53:47,144 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:53:47,144 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:53:47,152 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. 2023-04-16 00:53:47,152 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. started... 2023-04-16 00:53:47,152 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. as already flushing 2023-04-16 00:53:47,152 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. 2023-04-16 00:53:47,152 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:53:47,152 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:53:47,152 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:47,181 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:53:47,182 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:53:47,219 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:47,219 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:47,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:47,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:53:47,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:53:47,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:53:47,220 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:47,220 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:53:47,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:53:47,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:47,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:47,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:53:47,227 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:53:47,227 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7a79e431[Count = 0] remaining members to acquire global barrier 2023-04-16 00:53:47,227 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:53:47,227 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:47,229 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:47,229 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:47,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:47,229 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:53:47,229 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:53:47,229 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:47,230 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:53:47,229 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:53:47,233 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:53:47,233 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:47,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:47,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:53:47,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:53:47,233 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:53:47,233 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:53:47,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:53:47,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:53:47,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:47,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:47,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:53:47,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:47,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:47,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:53:47,242 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:53:47,242 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:53:47,242 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:53:47,242 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:53:47,243 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:53:47,243 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:53:47,245 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:47,245 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:47,245 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:47,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:47,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:47,245 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:53:47,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:53:47,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:53:47,246 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:53:47,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:53:47,247 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:47,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:47,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:53:47,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:47,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:53:47,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:47,249 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:47,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:47,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:53:47,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:47,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:47,253 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:53:47,253 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:53:47,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:53:47,253 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:53:47,253 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:53:47,253 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:53:47,253 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:53:47,253 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:53:47,253 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:47,254 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:47,254 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:47,254 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:47,254 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:47,254 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:47,254 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:53:47,253 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:53:47,254 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:53:47,254 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:47,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:53:47,296 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-16 00:53:47,357 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:53:47,358 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:53:47,367 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:53:47,386 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:53:47,388 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40044 deadline: 1681606487372, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:47,391 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:53:47,391 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39980 deadline: 1681606487385, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:47,391 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:53:47,392 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40000 deadline: 1681606487388, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:47,393 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:53:47,394 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:53:47,395 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40046 deadline: 1681606487388, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:47,395 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:53:47,395 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:53:47,395 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:53:47,396 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:53:47,396 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:53:47,396 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:53:47,397 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:47,405 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39990 deadline: 1681606487393, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:47,405 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:47,407 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:53:47,407 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:47,407 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:53:47,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:53:47,407 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:47,407 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:53:47,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:47,408 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:47,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:53:47,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:47,409 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:53:47,432 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:53:47,432 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:53:47,432 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:53:47,432 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:53:47,438 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:53:47,447 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:53:47,447 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:53:47,452 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. 2023-04-16 00:53:47,452 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. started... 2023-04-16 00:53:47,453 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. as already flushing 2023-04-16 00:53:47,453 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. 2023-04-16 00:53:47,453 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:53:47,453 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:53:47,453 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:47,453 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:53:47,453 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:53:47,456 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:47,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:47,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:53:47,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:53:47,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:53:47,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:53:47,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:47,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:47,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:53:47,460 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:47,461 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:53:47,461 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5d060730[Count = 0] remaining members to acquire global barrier 2023-04-16 00:53:47,461 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:53:47,461 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:47,461 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:47,461 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:53:47,463 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:47,463 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:47,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:47,463 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:53:47,463 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:53:47,463 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:53:47,465 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:47,465 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:53:47,465 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:53:47,465 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:53:47,492 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:53:47,492 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:53:47,492 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:53:47,492 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:53:47,492 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:53:47,492 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:53:47,492 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:53:47,496 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:47,496 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:53:47,496 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:47,496 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:47,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:47,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:53:47,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:53:47,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:47,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:53:47,497 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:53:47,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:53:47,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:47,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:47,499 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:47,501 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:47,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:53:47,505 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:53:47,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:53:47,506 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:47,506 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:47,506 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:47,506 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:53:47,506 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:53:47,505 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:53:47,505 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:53:47,506 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:47,518 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:47,518 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:47,518 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:47,506 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:53:47,515 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:53:47,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:53:47,519 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:53:47,519 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:53:47,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:53:47,520 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:53:47,620 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:53:47,621 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=59 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/.tmp/C/6ed04a0f0ac149249b0e67dbe6ef4bdd 2023-04-16 00:53:47,623 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:53:47,629 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:53:47,631 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:53:47,631 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:53:47,632 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:53:47,632 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:53:47,634 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:53:47,634 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:53:47,634 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:47,635 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:47,636 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:53:47,636 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:53:47,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:53:47,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:47,637 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:47,638 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:47,638 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/.tmp/A/04df6097d94a4008940de9cc7c096e60 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/A/04df6097d94a4008940de9cc7c096e60 2023-04-16 00:53:47,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:53:47,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:47,639 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:53:47,639 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:47,639 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:53:47,649 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/A/04df6097d94a4008940de9cc7c096e60, entries=200, sequenceid=59, filesize=13.9 K 2023-04-16 00:53:47,650 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:53:47,654 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/.tmp/B/6c92c274cda945e9adc21f14b3b3c391 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/B/6c92c274cda945e9adc21f14b3b3c391 2023-04-16 00:53:47,656 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:53:47,657 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:53:47,657 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:53:47,658 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:53:47,667 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:53:47,667 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:53:47,667 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. 2023-04-16 00:53:47,667 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. started... 2023-04-16 00:53:47,667 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. as already flushing 2023-04-16 00:53:47,668 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. 2023-04-16 00:53:47,668 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:53:47,668 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:53:47,668 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:47,668 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:53:47,668 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:53:47,673 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:47,673 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:47,674 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:53:47,674 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:47,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:47,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:53:47,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:53:47,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:53:47,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:53:47,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:47,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:47,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:53:47,685 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:53:47,686 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4defa7c3[Count = 0] remaining members to acquire global barrier 2023-04-16 00:53:47,686 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:53:47,686 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:47,687 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:47,687 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:47,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:47,687 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:53:47,687 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:53:47,688 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:53:47,688 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/B/6c92c274cda945e9adc21f14b3b3c391, entries=150, sequenceid=59, filesize=11.6 K 2023-04-16 00:53:47,688 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:47,688 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:53:47,691 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:53:47,691 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:47,691 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:53:47,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:47,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:53:47,691 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:53:47,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:53:47,698 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/.tmp/C/6ed04a0f0ac149249b0e67dbe6ef4bdd as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/C/6ed04a0f0ac149249b0e67dbe6ef4bdd 2023-04-16 00:53:47,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:53:47,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:53:47,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:47,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:47,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:53:47,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:47,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:47,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:53:47,707 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:53:47,707 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:53:47,707 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:53:47,707 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:53:47,707 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:53:47,707 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:53:47,709 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:47,709 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:47,709 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:47,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:47,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:53:47,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:53:47,709 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:53:47,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:47,710 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:47,710 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:53:47,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:53:47,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:53:47,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:47,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:47,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:53:47,712 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:47,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:47,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:47,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:53:47,715 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/C/6ed04a0f0ac149249b0e67dbe6ef4bdd, entries=150, sequenceid=59, filesize=11.6 K 2023-04-16 00:53:47,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:47,720 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:47,720 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:47,721 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:47,721 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:47,721 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:47,721 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=118.13 KB/120965 for c9039369968c422a52a2c7db83b430ba in 1282ms, sequenceid=59, compaction requested=true 2023-04-16 00:53:47,721 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:53:47,721 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for c9039369968c422a52a2c7db83b430ba: 2023-04-16 00:53:47,721 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:53:47,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:53:47,721 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:47,721 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:47,721 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:53:47,722 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:53:47,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:53:47,722 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:53:47,722 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:53:47,724 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:53:47,724 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:53:47,724 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:53:47,727 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 00:53:47,728 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-16 00:53:47,728 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-16 00:53:47,728 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:53:47,729 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 00:53:47,731 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 00:53:47,734 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-16 00:53:47,734 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37998 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-16 00:53:47,737 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HStore(1912): c9039369968c422a52a2c7db83b430ba/B is initiating minor compaction (all files) 2023-04-16 00:53:47,737 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HRegion(2259): Starting compaction of c9039369968c422a52a2c7db83b430ba/B in TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. 2023-04-16 00:53:47,737 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/B/bf4078543a914d21bb8934742f1d599a, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/B/430794e955ce43c284188aef09954da9, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/B/6c92c274cda945e9adc21f14b3b3c391] into tmpdir=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/.tmp, totalSize=34.8 K 2023-04-16 00:53:47,738 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting bf4078543a914d21bb8934742f1d599a, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=12, earliestPutTs=1681606424525 2023-04-16 00:53:47,739 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting 430794e955ce43c284188aef09954da9, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=47, earliestPutTs=1681606424738 2023-04-16 00:53:47,740 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting 6c92c274cda945e9adc21f14b3b3c391, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=59, earliestPutTs=1681606426068 2023-04-16 00:53:47,742 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HStore(1912): c9039369968c422a52a2c7db83b430ba/A is initiating minor compaction (all files) 2023-04-16 00:53:47,745 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of c9039369968c422a52a2c7db83b430ba/A in TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. 2023-04-16 00:53:47,745 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/A/5060fe90e1494ff1977278cbc188497d, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/A/b395a4b8fa144ace900151c416009e87, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/A/04df6097d94a4008940de9cc7c096e60] into tmpdir=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/.tmp, totalSize=37.1 K 2023-04-16 00:53:47,760 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.Compactor(207): Compacting 5060fe90e1494ff1977278cbc188497d, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=12, earliestPutTs=1681606424525 2023-04-16 00:53:47,763 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.Compactor(207): Compacting b395a4b8fa144ace900151c416009e87, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=47, earliestPutTs=1681606424738 2023-04-16 00:53:47,764 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.Compactor(207): Compacting 04df6097d94a4008940de9cc7c096e60, keycount=200, bloomtype=ROW, size=13.9 K, encoding=NONE, compression=NONE, seqNum=59, earliestPutTs=1681606426068 2023-04-16 00:53:47,794 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] throttle.PressureAwareThroughputController(145): c9039369968c422a52a2c7db83b430ba#B#compaction#9 average throughput is 1.31 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-16 00:53:47,819 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] throttle.PressureAwareThroughputController(145): c9039369968c422a52a2c7db83b430ba#A#compaction#10 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-16 00:53:47,833 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:53:47,841 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:53:47,848 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:53:47,868 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:53:47,889 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:53:47,889 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:53:47,889 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:53:47,892 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:53:47,892 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:53:47,895 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:47,895 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:47,896 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:53:47,896 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:47,897 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:53:47,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:53:47,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:47,898 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:47,898 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:53:47,898 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:47,901 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing c9039369968c422a52a2c7db83b430ba 3/3 column families, dataSize=131.55 KB heapSize=374.30 KB 2023-04-16 00:53:47,901 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c9039369968c422a52a2c7db83b430ba, store=A 2023-04-16 00:53:47,901 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 00:53:47,901 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c9039369968c422a52a2c7db83b430ba, store=B 2023-04-16 00:53:47,901 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 00:53:47,901 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c9039369968c422a52a2c7db83b430ba, store=C 2023-04-16 00:53:47,902 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(9158): Flush requested on c9039369968c422a52a2c7db83b430ba 2023-04-16 00:53:47,902 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 00:53:47,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:53:47,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:47,903 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:53:47,904 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:53:47,904 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:53:47,904 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:53:47,911 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:53:47,926 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:53:47,927 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:53:47,927 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. 2023-04-16 00:53:47,927 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. started... 2023-04-16 00:53:47,927 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. as already flushing 2023-04-16 00:53:47,928 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. 2023-04-16 00:53:47,927 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:53:47,928 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:53:47,928 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:53:47,928 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:47,928 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:53:47,929 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:53:47,932 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:47,932 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:47,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:47,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:53:47,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:53:47,934 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:47,934 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:53:47,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:53:47,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:53:47,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:47,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:47,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:53:47,942 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:53:47,942 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:53:47,942 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@35a8e242[Count = 0] remaining members to acquire global barrier 2023-04-16 00:53:47,942 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:47,947 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:47,947 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:53:47,947 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:47,947 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:47,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:47,948 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:53:47,948 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:53:47,948 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:53:47,949 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/.tmp/A/ffd4bea06da94cce9d5c37645615813f as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/A/ffd4bea06da94cce9d5c37645615813f 2023-04-16 00:53:47,951 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:53:47,951 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:53:47,951 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:53:47,951 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:47,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:47,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:53:47,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:53:47,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:53:47,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:53:47,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:47,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:47,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:53:47,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:47,957 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:53:47,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:47,957 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40000 deadline: 1681606487937, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:47,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:53:47,958 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:53:47,958 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:53:47,958 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:53:47,958 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:53:47,958 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:53:47,958 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:53:47,958 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/.tmp/B/82192c04cd374d1c8c0ef3b1d200daeb as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/B/82192c04cd374d1c8c0ef3b1d200daeb 2023-04-16 00:53:47,958 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:53:47,958 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40044 deadline: 1681606487939, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:47,959 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:53:47,959 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40046 deadline: 1681606487940, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:47,963 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:53:47,963 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39990 deadline: 1681606487944, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:47,964 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:53:47,964 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39980 deadline: 1681606487945, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:47,964 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:47,964 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:53:47,965 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:47,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:47,965 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:47,965 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:53:47,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:53:47,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:47,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:53:47,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:53:47,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:47,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:53:47,967 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:47,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:47,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:53:47,970 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:47,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:47,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:47,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:53:47,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:47,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:47,984 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:53:47,984 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:53:47,985 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:53:47,985 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:47,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:53:47,985 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:47,985 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:53:47,985 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:53:47,985 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:47,985 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:47,985 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:47,985 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:53:47,985 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:53:47,985 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:53:47,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:53:47,985 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:47,986 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:47,985 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:53:47,986 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:53:48,003 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=95 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/.tmp/A/b2f8b3bc86ca446baead5627f8710419 2023-04-16 00:53:48,006 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in c9039369968c422a52a2c7db83b430ba/B of c9039369968c422a52a2c7db83b430ba into 82192c04cd374d1c8c0ef3b1d200daeb(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 00:53:48,006 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for c9039369968c422a52a2c7db83b430ba: 2023-04-16 00:53:48,006 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in c9039369968c422a52a2c7db83b430ba/A of c9039369968c422a52a2c7db83b430ba into ffd4bea06da94cce9d5c37645615813f(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 00:53:48,006 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba., storeName=c9039369968c422a52a2c7db83b430ba/B, priority=13, startTime=1681606427727; duration=0sec 2023-04-16 00:53:48,006 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for c9039369968c422a52a2c7db83b430ba: 2023-04-16 00:53:48,006 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba., storeName=c9039369968c422a52a2c7db83b430ba/A, priority=13, startTime=1681606427724; duration=0sec 2023-04-16 00:53:48,007 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 00:53:48,007 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-16 00:53:48,009 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 00:53:48,012 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-16 00:53:48,012 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HStore(1912): c9039369968c422a52a2c7db83b430ba/C is initiating minor compaction (all files) 2023-04-16 00:53:48,012 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HRegion(2259): Starting compaction of c9039369968c422a52a2c7db83b430ba/C in TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. 2023-04-16 00:53:48,012 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/C/0da3dfc269354e279ba38e2eed98439d, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/C/cca93818a0814e2b8ad85e1628228b7b, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/C/6ed04a0f0ac149249b0e67dbe6ef4bdd] into tmpdir=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/.tmp, totalSize=34.8 K 2023-04-16 00:53:48,013 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting 0da3dfc269354e279ba38e2eed98439d, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=12, earliestPutTs=1681606424525 2023-04-16 00:53:48,015 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting cca93818a0814e2b8ad85e1628228b7b, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=47, earliestPutTs=1681606424738 2023-04-16 00:53:48,015 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting 6ed04a0f0ac149249b0e67dbe6ef4bdd, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=59, earliestPutTs=1681606426068 2023-04-16 00:53:48,050 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] throttle.PressureAwareThroughputController(145): c9039369968c422a52a2c7db83b430ba#C#compaction#12 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-16 00:53:48,083 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:53:48,086 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40044 deadline: 1681606488063, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:48,086 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:53:48,086 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:53:48,107 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40046 deadline: 1681606488067, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:48,107 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:53:48,107 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39990 deadline: 1681606488067, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:48,108 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:53:48,108 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:53:48,108 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39980 deadline: 1681606488081, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:48,111 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:53:48,112 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40000 deadline: 1681606488087, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:48,126 DEBUG [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(130): Registering adapter for the MetricRegistry: RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-04-16 00:53:48,127 INFO [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(134): Registering RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint Metrics about HBase RegionObservers 2023-04-16 00:53:48,170 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:53:48,181 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:53:48,184 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:53:48,184 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:53:48,184 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:53:48,185 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:53:48,185 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:53:48,187 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:48,187 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:48,194 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:48,194 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:53:48,194 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:53:48,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:53:48,195 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:48,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:48,195 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:53:48,195 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:48,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:53:48,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:48,196 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:53:48,196 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:53:48,196 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:53:48,196 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:53:48,196 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:53:48,197 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:53:48,198 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:53:48,198 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:53:48,199 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. 2023-04-16 00:53:48,199 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. started... 2023-04-16 00:53:48,200 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. as already flushing 2023-04-16 00:53:48,200 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. 2023-04-16 00:53:48,200 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:53:48,201 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:53:48,201 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:48,201 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:53:48,201 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:53:48,206 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:48,206 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:48,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:48,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:53:48,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:53:48,207 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:48,207 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:53:48,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:53:48,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:53:48,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:48,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:48,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:53:48,212 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:53:48,212 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@45c0aa[Count = 0] remaining members to acquire global barrier 2023-04-16 00:53:48,212 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:53:48,212 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:48,214 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:48,214 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:48,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:48,214 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:53:48,214 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:53:48,214 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:53:48,214 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:48,214 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:53:48,216 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:53:48,216 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:53:48,216 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:48,216 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:53:48,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:48,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:53:48,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:53:48,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:53:48,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:53:48,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:48,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:48,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:53:48,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:48,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:48,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:53:48,224 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:53:48,224 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:53:48,224 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:53:48,224 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:53:48,224 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:53:48,224 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:53:48,226 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:48,226 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:53:48,226 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:48,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:48,226 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:48,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:48,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:53:48,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:53:48,227 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:53:48,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:53:48,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:48,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:53:48,228 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:48,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:48,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:53:48,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:48,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:48,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:53:48,234 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:48,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:48,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:48,240 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:53:48,240 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:53:48,240 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:53:48,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:53:48,241 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:53:48,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:53:48,244 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:48,244 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:48,244 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:48,244 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:48,244 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:48,244 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:48,244 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:48,244 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:53:48,244 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:53:48,249 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:53:48,249 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:53:48,249 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:53:48,290 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:53:48,290 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40044 deadline: 1681606488289, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:48,312 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:53:48,313 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39990 deadline: 1681606488311, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:48,315 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:53:48,316 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39980 deadline: 1681606488315, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:48,317 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:53:48,318 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40046 deadline: 1681606488316, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:48,348 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:53:48,349 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40000 deadline: 1681606488346, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:48,349 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:53:48,350 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:53:48,357 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:53:48,360 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:53:48,360 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:53:48,360 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:53:48,360 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:53:48,361 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:53:48,361 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:53:48,362 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:48,362 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:48,363 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:48,363 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:53:48,364 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:53:48,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:53:48,364 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:48,364 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:53:48,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:48,367 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:48,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:53:48,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:48,367 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:53:48,368 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:53:48,368 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:53:48,368 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:53:48,368 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:53:48,369 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:53:48,370 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:53:48,370 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:53:48,370 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. 2023-04-16 00:53:48,371 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. started... 2023-04-16 00:53:48,371 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. as already flushing 2023-04-16 00:53:48,371 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. 2023-04-16 00:53:48,371 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:53:48,371 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:53:48,371 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:48,371 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:53:48,371 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:53:48,373 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:48,373 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:48,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:48,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:53:48,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:53:48,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:53:48,373 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:48,374 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:53:48,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:53:48,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:48,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:48,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:53:48,376 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:53:48,376 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6d177815[Count = 0] remaining members to acquire global barrier 2023-04-16 00:53:48,376 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:53:48,376 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:48,377 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:48,377 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:48,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:48,377 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:53:48,377 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:53:48,377 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:48,377 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:53:48,377 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:53:48,381 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:53:48,381 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:53:48,382 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:53:48,381 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:48,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:48,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:53:48,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:53:48,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:53:48,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:53:48,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:48,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:48,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:53:48,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:48,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:48,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:53:48,388 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:53:48,388 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:53:48,388 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:53:48,388 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:53:48,388 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:53:48,388 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:53:48,390 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:48,390 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:53:48,390 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:48,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:48,391 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:48,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:48,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:53:48,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:53:48,391 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:53:48,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:53:48,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:53:48,392 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:48,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:48,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:48,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:53:48,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:48,402 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:48,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:48,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:53:48,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:48,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:48,407 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:53:48,407 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:53:48,407 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:53:48,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:53:48,407 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:48,407 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:48,407 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:48,407 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:48,407 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:48,407 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:53:48,407 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:53:48,407 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:53:48,407 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:53:48,407 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:53:48,407 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:48,408 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:48,408 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:53:48,409 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:53:48,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:53:48,509 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:53:48,510 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:53:48,514 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:53:48,524 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:53:48,524 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:53:48,524 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:53:48,524 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:53:48,525 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:53:48,525 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:53:48,526 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:48,526 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:48,529 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:53:48,529 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:48,529 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:53:48,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:53:48,529 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:48,529 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:53:48,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:48,530 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:48,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:53:48,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:48,531 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:53:48,531 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:53:48,531 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:53:48,531 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:53:48,531 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:53:48,532 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:53:48,532 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:53:48,533 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:53:48,533 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. 2023-04-16 00:53:48,533 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. started... 2023-04-16 00:53:48,533 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. as already flushing 2023-04-16 00:53:48,533 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. 2023-04-16 00:53:48,533 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:53:48,533 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:53:48,533 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:48,533 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:53:48,533 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:53:48,535 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:48,535 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:48,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:48,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:53:48,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:53:48,535 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:48,535 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:53:48,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:53:48,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:53:48,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:48,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:48,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:53:48,538 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:53:48,538 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:53:48,538 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:48,538 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5d761bcb[Count = 0] remaining members to acquire global barrier 2023-04-16 00:53:48,540 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:48,540 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:48,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:48,540 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:48,540 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:53:48,540 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:53:48,540 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:53:48,540 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:53:48,542 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:53:48,542 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:48,542 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:53:48,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:48,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:53:48,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:53:48,542 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:53:48,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:53:48,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:53:48,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:48,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:48,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:53:48,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:48,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:48,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:53:48,546 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:53:48,546 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:53:48,546 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:53:48,546 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:53:48,546 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:53:48,546 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:53:48,552 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:48,552 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:48,552 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:48,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:48,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:53:48,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:53:48,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:48,552 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:53:48,553 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:53:48,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:53:48,554 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:48,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:53:48,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:48,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:48,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:53:48,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:48,557 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:48,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:48,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:53:48,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:48,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:48,562 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:53:48,562 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:53:48,562 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:53:48,562 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:53:48,562 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:53:48,562 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:48,562 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:48,562 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:48,562 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:48,562 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:48,562 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:53:48,562 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:53:48,562 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:48,562 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:53:48,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:53:48,564 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:53:48,563 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:48,565 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:53:48,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:53:48,582 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/.tmp/C/d68816e41a76482f91678e90e7f7411b as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/C/d68816e41a76482f91678e90e7f7411b 2023-04-16 00:53:48,594 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=95 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/.tmp/B/03fc509a50fd486a9e529b8686d7f030 2023-04-16 00:53:48,596 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:53:48,596 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40044 deadline: 1681606488595, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:48,599 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in c9039369968c422a52a2c7db83b430ba/C of c9039369968c422a52a2c7db83b430ba into d68816e41a76482f91678e90e7f7411b(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 00:53:48,601 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for c9039369968c422a52a2c7db83b430ba: 2023-04-16 00:53:48,601 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba., storeName=c9039369968c422a52a2c7db83b430ba/C, priority=13, startTime=1681606427731; duration=0sec 2023-04-16 00:53:48,601 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 00:53:48,618 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:53:48,618 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39990 deadline: 1681606488617, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:48,631 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:53:48,632 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39980 deadline: 1681606488620, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:48,632 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:53:48,632 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40046 deadline: 1681606488621, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:48,653 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.09 KB at sequenceid=95 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/.tmp/C/2b2651e29329451a84a5318c553ae69c 2023-04-16 00:53:48,654 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:53:48,655 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40000 deadline: 1681606488652, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:48,664 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:53:48,665 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:53:48,665 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/.tmp/A/b2f8b3bc86ca446baead5627f8710419 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/A/b2f8b3bc86ca446baead5627f8710419 2023-04-16 00:53:48,669 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:53:48,672 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:53:48,672 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:53:48,672 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:53:48,672 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:53:48,676 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:53:48,676 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:53:48,677 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:48,678 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:48,680 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:48,680 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:53:48,681 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:53:48,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:53:48,681 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:48,681 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:53:48,682 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/A/b2f8b3bc86ca446baead5627f8710419, entries=250, sequenceid=95, filesize=16.2 K 2023-04-16 00:53:48,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:48,701 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:48,701 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/.tmp/B/03fc509a50fd486a9e529b8686d7f030 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/B/03fc509a50fd486a9e529b8686d7f030 2023-04-16 00:53:48,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:53:48,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:48,702 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:53:48,717 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:53:48,717 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:53:48,717 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:53:48,719 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:53:48,719 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:53:48,719 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:53:48,720 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. 2023-04-16 00:53:48,720 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. started... 2023-04-16 00:53:48,720 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. as already flushing 2023-04-16 00:53:48,720 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. 2023-04-16 00:53:48,720 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:53:48,720 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:53:48,720 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:48,721 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:53:48,721 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:53:48,723 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:48,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:48,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:53:48,724 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:48,725 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:48,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:53:48,725 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:53:48,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:53:48,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:53:48,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:48,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:48,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:53:48,728 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:53:48,729 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2f3e48d5[Count = 0] remaining members to acquire global barrier 2023-04-16 00:53:48,729 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:53:48,729 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:48,732 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:48,732 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:53:48,732 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:48,732 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:48,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:48,732 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:53:48,733 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:53:48,733 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:53:48,734 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:48,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:48,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:53:48,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:53:48,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:53:48,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:53:48,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:48,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:48,736 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:53:48,737 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:53:48,737 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:53:48,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:53:48,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:48,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:48,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:53:48,740 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:53:48,740 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:53:48,740 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:53:48,740 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:53:48,740 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:53:48,740 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:53:48,740 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/B/03fc509a50fd486a9e529b8686d7f030, entries=150, sequenceid=95, filesize=11.6 K 2023-04-16 00:53:48,742 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:48,742 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:53:48,742 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:48,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:48,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:53:48,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:53:48,743 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:48,743 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/.tmp/C/2b2651e29329451a84a5318c553ae69c as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/C/2b2651e29329451a84a5318c553ae69c 2023-04-16 00:53:48,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:48,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:53:48,743 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:53:48,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:53:48,743 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:48,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:48,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:48,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:53:48,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:48,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:48,752 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:48,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:53:48,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:48,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:48,757 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:48,757 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:48,757 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:53:48,757 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:53:48,757 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:53:48,757 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:53:48,758 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:53:48,757 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:53:48,758 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:53:48,758 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:53:48,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:53:48,759 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:48,759 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:48,759 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:48,760 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:48,760 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:48,762 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:53:48,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:53:48,770 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/C/2b2651e29329451a84a5318c553ae69c, entries=150, sequenceid=95, filesize=11.6 K 2023-04-16 00:53:48,774 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~131.55 KB/134705, heapSize ~374.26 KB/383240, currentSize=53.67 KB/54960 for c9039369968c422a52a2c7db83b430ba in 874ms, sequenceid=95, compaction requested=false 2023-04-16 00:53:48,774 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for c9039369968c422a52a2c7db83b430ba: 2023-04-16 00:53:48,858 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:53:48,859 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:53:48,863 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:53:48,866 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:53:48,866 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:53:48,866 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:53:48,866 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:53:48,869 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:53:48,869 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:53:48,870 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:48,870 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:48,873 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:53:48,873 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:53:48,873 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:48,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:53:48,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:48,874 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:48,874 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:53:48,874 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:48,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:53:48,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:48,875 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:53:48,888 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:53:48,888 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:53:48,888 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:53:48,888 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:53:48,890 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:53:48,900 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:53:48,900 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:53:48,902 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. 2023-04-16 00:53:48,902 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. started... 2023-04-16 00:53:48,903 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing c9039369968c422a52a2c7db83b430ba 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-16 00:53:48,906 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c9039369968c422a52a2c7db83b430ba, store=A 2023-04-16 00:53:48,906 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:53:48,906 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c9039369968c422a52a2c7db83b430ba, store=B 2023-04-16 00:53:48,906 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:53:48,907 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c9039369968c422a52a2c7db83b430ba, store=C 2023-04-16 00:53:48,907 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:53:49,107 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(9158): Flush requested on c9039369968c422a52a2c7db83b430ba 2023-04-16 00:53:49,107 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. as already flushing 2023-04-16 00:53:49,160 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681606429143, max timestamp=1681606429154Num uniques 859; 2023-04-16 00:53:49,160 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-16 00:53:49,161 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681606429143, max timestamp=1681606429155Num uniques 859; 2023-04-16 00:53:49,161 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-16 00:53:49,165 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1681606429143, max timestamp=1681606429154Num uniques 859; 2023-04-16 00:53:49,165 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=6870, suffix heap size=117064,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=150 2023-04-16 00:53:49,189 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:53:49,189 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40044 deadline: 1681606489183, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:49,190 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:53:49,190 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:53:49,190 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39990 deadline: 1681606489184, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:49,190 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40046 deadline: 1681606489184, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:49,190 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:53:49,190 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40000 deadline: 1681606489189, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:49,192 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:53:49,192 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39980 deadline: 1681606489189, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:49,293 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:53:49,293 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40044 deadline: 1681606489291, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:49,293 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:53:49,294 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40046 deadline: 1681606489291, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:49,294 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:53:49,295 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:53:49,295 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40000 deadline: 1681606489292, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:49,295 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39990 deadline: 1681606489291, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:49,298 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:53:49,298 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39980 deadline: 1681606489293, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:49,351 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=109 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/.tmp/A/5cbc5ead12464029a17eec293877e462 2023-04-16 00:53:49,397 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=109 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/.tmp/B/b1725dab9e3e485d819a349709860508 2023-04-16 00:53:49,443 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=109 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/.tmp/C/fd0d64e0bc054fa19000e0649788fe0f 2023-04-16 00:53:49,453 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/.tmp/A/5cbc5ead12464029a17eec293877e462 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/A/5cbc5ead12464029a17eec293877e462 2023-04-16 00:53:49,462 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/A/5cbc5ead12464029a17eec293877e462, entries=100, sequenceid=109, filesize=9.3 K 2023-04-16 00:53:49,466 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/.tmp/B/b1725dab9e3e485d819a349709860508 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/B/b1725dab9e3e485d819a349709860508 2023-04-16 00:53:49,478 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/B/b1725dab9e3e485d819a349709860508, entries=100, sequenceid=109, filesize=9.3 K 2023-04-16 00:53:49,481 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/.tmp/C/fd0d64e0bc054fa19000e0649788fe0f as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/C/fd0d64e0bc054fa19000e0649788fe0f 2023-04-16 00:53:49,490 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/C/fd0d64e0bc054fa19000e0649788fe0f, entries=100, sequenceid=109, filesize=9.3 K 2023-04-16 00:53:49,493 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.08 KB/130125 for c9039369968c422a52a2c7db83b430ba in 589ms, sequenceid=109, compaction requested=true 2023-04-16 00:53:49,493 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for c9039369968c422a52a2c7db83b430ba: 2023-04-16 00:53:49,493 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. 2023-04-16 00:53:49,493 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:53:49,494 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:53:49,494 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:49,494 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:53:49,494 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:53:49,496 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:49,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:49,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:53:49,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:53:49,496 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:49,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:53:49,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:53:49,497 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:49,497 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:53:49,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:49,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:49,500 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(9158): Flush requested on c9039369968c422a52a2c7db83b430ba 2023-04-16 00:53:49,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:53:49,500 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing c9039369968c422a52a2c7db83b430ba 3/3 column families, dataSize=133.78 KB heapSize=384.07 KB 2023-04-16 00:53:49,500 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c9039369968c422a52a2c7db83b430ba, store=A 2023-04-16 00:53:49,500 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 00:53:49,500 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c9039369968c422a52a2c7db83b430ba, store=B 2023-04-16 00:53:49,501 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 00:53:49,501 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c9039369968c422a52a2c7db83b430ba, store=C 2023-04-16 00:53:49,501 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 00:53:49,502 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:53:49,502 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1509d08[Count = 0] remaining members to acquire global barrier 2023-04-16 00:53:49,502 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:53:49,502 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:49,504 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:49,504 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:49,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:49,504 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:53:49,504 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:49,504 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:53:49,504 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:53:49,504 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:53:49,507 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:53:49,507 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:49,507 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:53:49,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:49,507 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:53:49,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:53:49,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:53:49,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:53:49,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:53:49,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:49,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:49,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:53:49,516 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:53:49,516 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39980 deadline: 1681606489507, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:49,517 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:53:49,517 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40044 deadline: 1681606489509, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:49,518 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:53:49,518 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40000 deadline: 1681606489511, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:49,518 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:53:49,518 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40046 deadline: 1681606489511, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:49,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:49,524 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:53:49,525 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39990 deadline: 1681606489524, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:49,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:49,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:53:49,530 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:53:49,530 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:53:49,530 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:53:49,530 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:53:49,531 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:53:49,531 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:53:49,533 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:49,533 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:53:49,533 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:49,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:49,533 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:53:49,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:53:49,534 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:49,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:49,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:49,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:53:49,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:53:49,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:53:49,537 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:49,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:49,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:53:49,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:49,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:49,538 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:49,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:53:49,542 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=146 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/.tmp/A/2626362e776b4e9c9eb512a0062c34c7 2023-04-16 00:53:49,544 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:53:49,544 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:49,544 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:53:49,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:53:49,544 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:53:49,544 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:49,550 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:49,550 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:53:49,550 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:53:49,544 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:53:49,550 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:49,550 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:53:49,550 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:53:49,550 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:53:49,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:53:49,550 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:49,550 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:49,550 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:49,550 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:53:49,614 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=146 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/.tmp/B/fd2f71d62d174dc1867610af432cfd3c 2023-04-16 00:53:49,626 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:53:49,626 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39980 deadline: 1681606489619, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:49,626 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:53:49,627 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:53:49,627 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40000 deadline: 1681606489619, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:49,627 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40044 deadline: 1681606489620, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:49,627 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:53:49,627 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40046 deadline: 1681606489626, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:49,632 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:53:49,632 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39990 deadline: 1681606489630, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:49,649 WARN [DataStreamer for file /user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/.tmp/C/11757a04a9cf417787c1a36a31497de0 block BP-1873989735-172.31.14.131-1681606414745:blk_1073741858_1034] hdfs.DataStreamer(982): Caught exception java.lang.InterruptedException at java.lang.Object.wait(Native Method) at java.lang.Thread.join(Thread.java:1257) at java.lang.Thread.join(Thread.java:1331) at org.apache.hadoop.hdfs.DataStreamer.closeResponder(DataStreamer.java:980) at org.apache.hadoop.hdfs.DataStreamer.closeInternal(DataStreamer.java:844) at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:840) 2023-04-16 00:53:49,650 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:53:49,651 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:53:49,652 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.33 KB at sequenceid=146 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/.tmp/C/11757a04a9cf417787c1a36a31497de0 2023-04-16 00:53:49,654 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:53:49,656 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:53:49,656 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:53:49,656 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:53:49,656 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:53:49,656 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:53:49,656 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:53:49,657 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:49,657 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:49,659 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:53:49,659 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:49,659 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:53:49,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:53:49,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:49,661 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:49,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:53:49,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:49,661 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:53:49,661 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:49,662 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:53:49,662 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/.tmp/A/2626362e776b4e9c9eb512a0062c34c7 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/A/2626362e776b4e9c9eb512a0062c34c7 2023-04-16 00:53:49,664 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:53:49,664 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:53:49,664 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:53:49,664 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:53:49,664 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:53:49,668 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:53:49,668 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:53:49,668 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. 2023-04-16 00:53:49,676 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. started... 2023-04-16 00:53:49,676 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. as already flushing 2023-04-16 00:53:49,676 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. 2023-04-16 00:53:49,676 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:53:49,676 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:53:49,676 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:49,676 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:53:49,676 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:53:49,678 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:49,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:49,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:53:49,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:53:49,684 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:49,684 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/A/2626362e776b4e9c9eb512a0062c34c7, entries=200, sequenceid=146, filesize=14.1 K 2023-04-16 00:53:49,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:53:49,685 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:49,685 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:53:49,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:53:49,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:49,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:49,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:53:49,686 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/.tmp/B/fd2f71d62d174dc1867610af432cfd3c as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/B/fd2f71d62d174dc1867610af432cfd3c 2023-04-16 00:53:49,687 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:53:49,687 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@57519ae7[Count = 0] remaining members to acquire global barrier 2023-04-16 00:53:49,687 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:53:49,687 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:49,689 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:49,689 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:49,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:49,689 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:53:49,689 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:53:49,689 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:53:49,689 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:49,689 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:53:49,694 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:53:49,694 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:49,694 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:53:49,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:49,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:53:49,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:53:49,694 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:53:49,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:53:49,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:53:49,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:49,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:49,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:53:49,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:49,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:49,701 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/B/fd2f71d62d174dc1867610af432cfd3c, entries=150, sequenceid=146, filesize=11.8 K 2023-04-16 00:53:49,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:53:49,704 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:53:49,704 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:53:49,704 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:53:49,704 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:53:49,704 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:53:49,704 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:53:49,706 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:49,706 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:53:49,706 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:49,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:49,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:53:49,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:53:49,706 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:49,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:49,707 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:53:49,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:53:49,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:53:49,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:49,708 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:49,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:49,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:53:49,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:49,710 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:49,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:49,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:53:49,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:49,713 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/.tmp/C/11757a04a9cf417787c1a36a31497de0 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/C/11757a04a9cf417787c1a36a31497de0 2023-04-16 00:53:49,721 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:53:49,721 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:53:49,721 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:49,722 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:53:49,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:53:49,722 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:53:49,722 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:53:49,722 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:53:49,722 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:49,722 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:49,722 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:49,722 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:49,722 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:49,722 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:49,722 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:53:49,722 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:53:49,722 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:53:49,722 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:53:49,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:53:49,729 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/C/11757a04a9cf417787c1a36a31497de0, entries=150, sequenceid=146, filesize=11.8 K 2023-04-16 00:53:49,731 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~133.78 KB/136995, heapSize ~384.02 KB/393240, currentSize=46.96 KB/48090 for c9039369968c422a52a2c7db83b430ba in 231ms, sequenceid=146, compaction requested=true 2023-04-16 00:53:49,731 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for c9039369968c422a52a2c7db83b430ba: 2023-04-16 00:53:49,731 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 00:53:49,731 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-16 00:53:49,731 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-16 00:53:49,731 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 00:53:49,732 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 00:53:49,734 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 52523 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-16 00:53:49,734 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HStore(1912): c9039369968c422a52a2c7db83b430ba/A is initiating minor compaction (all files) 2023-04-16 00:53:49,734 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of c9039369968c422a52a2c7db83b430ba/A in TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. 2023-04-16 00:53:49,734 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/A/ffd4bea06da94cce9d5c37645615813f, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/A/b2f8b3bc86ca446baead5627f8710419, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/A/5cbc5ead12464029a17eec293877e462, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/A/2626362e776b4e9c9eb512a0062c34c7] into tmpdir=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/.tmp, totalSize=51.3 K 2023-04-16 00:53:49,735 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 45453 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-16 00:53:49,735 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HStore(1912): c9039369968c422a52a2c7db83b430ba/B is initiating minor compaction (all files) 2023-04-16 00:53:49,735 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HRegion(2259): Starting compaction of c9039369968c422a52a2c7db83b430ba/B in TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. 2023-04-16 00:53:49,735 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.Compactor(207): Compacting ffd4bea06da94cce9d5c37645615813f, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=59, earliestPutTs=1681606426068 2023-04-16 00:53:49,736 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.Compactor(207): Compacting b2f8b3bc86ca446baead5627f8710419, keycount=250, bloomtype=ROW, size=16.2 K, encoding=NONE, compression=NONE, seqNum=95, earliestPutTs=1681606426583 2023-04-16 00:53:49,737 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.Compactor(207): Compacting 5cbc5ead12464029a17eec293877e462, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=109, earliestPutTs=1681606427938 2023-04-16 00:53:49,738 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.Compactor(207): Compacting 2626362e776b4e9c9eb512a0062c34c7, keycount=200, bloomtype=ROW, size=14.1 K, encoding=NONE, compression=NONE, seqNum=146, earliestPutTs=1681606429181 2023-04-16 00:53:49,738 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/B/82192c04cd374d1c8c0ef3b1d200daeb, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/B/03fc509a50fd486a9e529b8686d7f030, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/B/b1725dab9e3e485d819a349709860508, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/B/fd2f71d62d174dc1867610af432cfd3c] into tmpdir=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/.tmp, totalSize=44.4 K 2023-04-16 00:53:49,739 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting 82192c04cd374d1c8c0ef3b1d200daeb, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=59, earliestPutTs=1681606426068 2023-04-16 00:53:49,740 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting 03fc509a50fd486a9e529b8686d7f030, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=95, earliestPutTs=1681606426583 2023-04-16 00:53:49,741 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting b1725dab9e3e485d819a349709860508, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=109, earliestPutTs=1681606427938 2023-04-16 00:53:49,741 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting fd2f71d62d174dc1867610af432cfd3c, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=146, earliestPutTs=1681606429181 2023-04-16 00:53:49,786 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] throttle.PressureAwareThroughputController(145): c9039369968c422a52a2c7db83b430ba#A#compaction#21 average throughput is 1.09 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-16 00:53:49,798 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] throttle.PressureAwareThroughputController(145): c9039369968c422a52a2c7db83b430ba#B#compaction#22 average throughput is 1.31 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-16 00:53:49,822 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:53:49,824 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:53:49,827 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:53:49,840 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(9158): Flush requested on c9039369968c422a52a2c7db83b430ba 2023-04-16 00:53:49,840 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing c9039369968c422a52a2c7db83b430ba 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-16 00:53:49,840 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c9039369968c422a52a2c7db83b430ba, store=A 2023-04-16 00:53:49,840 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:53:49,841 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c9039369968c422a52a2c7db83b430ba, store=B 2023-04-16 00:53:49,841 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:53:49,841 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c9039369968c422a52a2c7db83b430ba, store=C 2023-04-16 00:53:49,841 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:53:49,845 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:53:49,846 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:53:49,846 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:53:49,846 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:53:49,852 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:53:49,852 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:53:49,854 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:49,854 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:49,866 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:49,866 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:53:49,866 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:53:49,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:53:49,867 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:49,867 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:53:49,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:49,869 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:49,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:53:49,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:49,871 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:53:49,873 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/.tmp/A/9bfdf85328814f7d9f54fd8988643218 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/A/9bfdf85328814f7d9f54fd8988643218 2023-04-16 00:53:49,876 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:53:49,876 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:53:49,876 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:53:49,876 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:53:49,876 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:53:49,884 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in c9039369968c422a52a2c7db83b430ba/A of c9039369968c422a52a2c7db83b430ba into 9bfdf85328814f7d9f54fd8988643218(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 00:53:49,884 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for c9039369968c422a52a2c7db83b430ba: 2023-04-16 00:53:49,884 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba., storeName=c9039369968c422a52a2c7db83b430ba/A, priority=12, startTime=1681606429731; duration=0sec 2023-04-16 00:53:49,885 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 00:53:49,885 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-16 00:53:49,889 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:53:49,898 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. 2023-04-16 00:53:49,898 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:53:49,898 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 45453 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-16 00:53:49,898 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. started... 2023-04-16 00:53:49,898 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HStore(1912): c9039369968c422a52a2c7db83b430ba/C is initiating minor compaction (all files) 2023-04-16 00:53:49,898 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. as already flushing 2023-04-16 00:53:49,898 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. 2023-04-16 00:53:49,898 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of c9039369968c422a52a2c7db83b430ba/C in TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. 2023-04-16 00:53:49,898 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:53:49,899 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:53:49,899 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:49,899 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/C/d68816e41a76482f91678e90e7f7411b, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/C/2b2651e29329451a84a5318c553ae69c, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/C/fd0d64e0bc054fa19000e0649788fe0f, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/C/11757a04a9cf417787c1a36a31497de0] into tmpdir=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/.tmp, totalSize=44.4 K 2023-04-16 00:53:49,899 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.Compactor(207): Compacting d68816e41a76482f91678e90e7f7411b, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=59, earliestPutTs=1681606426068 2023-04-16 00:53:49,900 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.Compactor(207): Compacting 2b2651e29329451a84a5318c553ae69c, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=95, earliestPutTs=1681606426583 2023-04-16 00:53:49,900 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.Compactor(207): Compacting fd0d64e0bc054fa19000e0649788fe0f, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=109, earliestPutTs=1681606427938 2023-04-16 00:53:49,901 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.Compactor(207): Compacting 11757a04a9cf417787c1a36a31497de0, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=146, earliestPutTs=1681606429181 2023-04-16 00:53:49,907 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:53:49,909 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:53:49,913 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:49,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:49,914 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:49,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:53:49,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:53:49,914 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:49,915 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:53:49,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:53:49,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:53:49,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:49,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:49,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:53:49,918 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=157 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/.tmp/A/c60ca2650c964955988021e770d8da8f 2023-04-16 00:53:49,919 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:53:49,919 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3a946b85[Count = 0] remaining members to acquire global barrier 2023-04-16 00:53:49,919 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:53:49,920 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:49,921 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:49,921 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:49,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:49,921 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:53:49,921 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:53:49,921 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:53:49,921 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:49,921 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:53:49,925 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:53:49,925 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:53:49,925 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:53:49,925 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:49,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:49,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:53:49,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:53:49,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:53:49,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:53:49,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:49,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:49,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:53:49,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:49,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:49,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:53:49,930 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:53:49,930 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:53:49,930 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:53:49,930 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:53:49,931 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:53:49,931 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:53:49,933 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:49,933 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:49,933 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:53:49,933 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:49,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:49,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:49,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:53:49,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:53:49,934 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:53:49,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:53:49,934 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:49,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:53:49,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:49,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:49,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:53:49,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:49,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:49,937 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:49,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:53:49,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:49,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:49,940 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:53:49,940 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:53:49,940 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:49,940 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:53:49,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:53:49,940 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:49,940 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:53:49,940 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:53:49,940 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:49,940 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:49,940 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:49,940 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:49,940 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:49,940 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:53:49,941 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:53:49,941 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:53:49,941 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:53:49,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:53:49,949 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:53:49,950 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40044 deadline: 1681606489928, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:49,950 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:53:49,950 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40000 deadline: 1681606489931, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:49,950 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681606429912, max timestamp=1681606429943Num uniques 859; 2023-04-16 00:53:49,951 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681606429912, max timestamp=1681606429943Num uniques 859; 2023-04-16 00:53:49,951 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-16 00:53:49,951 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-16 00:53:49,951 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] throttle.PressureAwareThroughputController(145): c9039369968c422a52a2c7db83b430ba#C#compaction#25 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-16 00:53:49,953 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1681606429912, max timestamp=1681606429931Num uniques 859; 2023-04-16 00:53:49,953 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=6870, suffix heap size=117064,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=150 2023-04-16 00:53:49,989 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=157 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/.tmp/B/376f37c0b97947a1aabe342a7d2e924d 2023-04-16 00:53:50,023 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:53:50,023 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39980 deadline: 1681606490014, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:50,030 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:53:50,030 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39990 deadline: 1681606490018, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:50,031 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:53:50,031 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40046 deadline: 1681606490018, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:50,049 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:53:50,050 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:53:50,054 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:53:50,055 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40044 deadline: 1681606490052, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:50,056 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:53:50,062 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:53:50,062 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40000 deadline: 1681606490055, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:50,064 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:53:50,064 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:53:50,064 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:53:50,064 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:53:50,064 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:53:50,065 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:53:50,065 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:50,065 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:50,068 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:53:50,068 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:53:50,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:53:50,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:50,069 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:50,069 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:50,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:53:50,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:50,069 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:53:50,071 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:50,071 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:53:50,080 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:53:50,080 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:53:50,080 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:53:50,086 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:53:50,086 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:53:50,089 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:53:50,089 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:53:50,094 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. 2023-04-16 00:53:50,094 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. started... 2023-04-16 00:53:50,096 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. as already flushing 2023-04-16 00:53:50,098 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. 2023-04-16 00:53:50,098 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:53:50,098 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:53:50,098 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:50,098 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:53:50,098 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:53:50,101 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:50,101 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:50,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:50,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:53:50,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:53:50,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:53:50,103 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:50,103 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:53:50,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:53:50,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:50,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:50,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:53:50,106 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:53:50,106 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6731e4e2[Count = 0] remaining members to acquire global barrier 2023-04-16 00:53:50,106 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:53:50,106 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:50,108 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:50,108 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:50,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:50,109 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:53:50,109 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:53:50,109 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:53:50,110 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:50,110 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:53:50,112 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:50,112 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:53:50,112 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:53:50,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:50,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:53:50,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:53:50,112 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:53:50,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:53:50,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:53:50,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:50,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:50,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:53:50,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:50,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:50,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:53:50,116 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:53:50,116 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:53:50,116 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:53:50,116 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:53:50,116 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:53:50,116 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:53:50,120 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:50,120 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:50,121 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:50,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:50,121 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:53:50,121 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:53:50,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:53:50,121 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:50,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:50,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:50,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:53:50,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:53:50,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:53:50,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:50,123 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:50,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:53:50,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:50,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:50,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:53:50,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:50,129 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:53:50,129 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:53:50,130 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:53:50,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:53:50,130 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:53:50,130 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:53:50,130 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39980 deadline: 1681606490125, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:50,130 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:53:50,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:53:50,132 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:53:50,132 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:53:50,132 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:53:50,133 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:53:50,133 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:53:50,136 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:50,136 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:50,136 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:50,136 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:50,136 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:50,136 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:50,136 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:50,141 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:53:50,141 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39990 deadline: 1681606490133, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:50,141 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:53:50,141 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40046 deadline: 1681606490133, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:50,233 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:53:50,234 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:53:50,237 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:53:50,238 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:53:50,238 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:53:50,238 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:53:50,239 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:53:50,239 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:53:50,239 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:53:50,240 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:50,240 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:50,243 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:53:50,243 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:50,243 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:53:50,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:53:50,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:50,243 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:50,243 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:53:50,244 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:50,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:53:50,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:50,245 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:53:50,256 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:53:50,256 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:53:50,256 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:53:50,256 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:53:50,256 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:53:50,259 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:53:50,260 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40044 deadline: 1681606490257, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:50,264 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:53:50,264 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:53:50,264 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. 2023-04-16 00:53:50,265 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. started... 2023-04-16 00:53:50,265 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. as already flushing 2023-04-16 00:53:50,265 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. 2023-04-16 00:53:50,265 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:53:50,265 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:53:50,265 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:50,265 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:53:50,265 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:53:50,270 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:50,270 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:50,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:50,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:53:50,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:53:50,270 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:50,270 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:53:50,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:53:50,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:53:50,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:50,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:50,273 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:53:50,273 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40000 deadline: 1681606490268, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:50,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:53:50,275 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:53:50,275 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@620c8dd1[Count = 0] remaining members to acquire global barrier 2023-04-16 00:53:50,275 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:53:50,275 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:50,278 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:50,278 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:50,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:50,279 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:53:50,279 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:53:50,279 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:53:50,280 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:50,280 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:53:50,281 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:50,281 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:53:50,281 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:53:50,282 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:53:50,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:50,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:53:50,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:53:50,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:53:50,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:53:50,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:50,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:50,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:53:50,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:50,286 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/.tmp/B/b9d2f4ed4a644dcaa49ae82bb4bb1d6c as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/B/b9d2f4ed4a644dcaa49ae82bb4bb1d6c 2023-04-16 00:53:50,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:50,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:53:50,290 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:53:50,290 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:53:50,290 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:53:50,290 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:53:50,290 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:53:50,290 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:53:50,293 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:50,293 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:50,293 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:50,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:50,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:53:50,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:53:50,293 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:53:50,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:50,294 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:50,294 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:53:50,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:53:50,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:53:50,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:50,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:50,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:53:50,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:50,296 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:50,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:50,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:53:50,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:50,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:50,305 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:53:50,305 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:53:50,305 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:50,305 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:53:50,305 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:50,305 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:53:50,305 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:53:50,305 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:50,306 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:50,306 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:50,306 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:50,306 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:53:50,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:53:50,305 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:53:50,306 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:53:50,306 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:53:50,306 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:50,306 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:53:50,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:53:50,306 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in c9039369968c422a52a2c7db83b430ba/B of c9039369968c422a52a2c7db83b430ba into b9d2f4ed4a644dcaa49ae82bb4bb1d6c(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 00:53:50,306 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for c9039369968c422a52a2c7db83b430ba: 2023-04-16 00:53:50,307 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba., storeName=c9039369968c422a52a2c7db83b430ba/B, priority=12, startTime=1681606429731; duration=0sec 2023-04-16 00:53:50,307 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 00:53:50,336 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:53:50,336 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39980 deadline: 1681606490333, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:50,347 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:53:50,348 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40046 deadline: 1681606490343, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:50,349 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:53:50,349 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39990 deadline: 1681606490344, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:50,405 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/.tmp/C/e7ef5613df4745038a32473837327c7b as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/C/e7ef5613df4745038a32473837327c7b 2023-04-16 00:53:50,407 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:53:50,408 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:53:50,411 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:53:50,419 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:53:50,419 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:53:50,419 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:53:50,419 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:53:50,419 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:53:50,419 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:53:50,419 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in c9039369968c422a52a2c7db83b430ba/C of c9039369968c422a52a2c7db83b430ba into e7ef5613df4745038a32473837327c7b(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 00:53:50,420 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for c9039369968c422a52a2c7db83b430ba: 2023-04-16 00:53:50,420 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba., storeName=c9039369968c422a52a2c7db83b430ba/C, priority=12, startTime=1681606429732; duration=0sec 2023-04-16 00:53:50,420 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 00:53:50,420 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:50,420 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:50,421 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:53:50,421 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:50,421 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:53:50,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:53:50,422 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:50,422 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:53:50,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:50,422 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:50,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:53:50,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:50,423 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:53:50,425 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:53:50,425 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:53:50,425 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:53:50,425 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:53:50,426 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:53:50,428 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:53:50,432 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. 2023-04-16 00:53:50,432 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. started... 2023-04-16 00:53:50,432 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:53:50,432 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. as already flushing 2023-04-16 00:53:50,432 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. 2023-04-16 00:53:50,432 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:53:50,433 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:53:50,433 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:50,433 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:53:50,433 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:53:50,436 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:50,436 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:50,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:50,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:53:50,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:53:50,436 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:50,436 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:53:50,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:53:50,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:53:50,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:50,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:50,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:53:50,438 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:53:50,438 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@39d1b5d5[Count = 0] remaining members to acquire global barrier 2023-04-16 00:53:50,438 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:53:50,439 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:50,441 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:50,441 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:50,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:50,441 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:53:50,441 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:53:50,441 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:53:50,442 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:50,442 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:53:50,443 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:53:50,443 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:53:50,444 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:53:50,443 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:50,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:50,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:53:50,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:53:50,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:53:50,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:53:50,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:50,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:50,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:53:50,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:50,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:50,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:53:50,446 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:53:50,446 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:53:50,446 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:53:50,446 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:53:50,447 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:53:50,447 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:53:50,449 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:50,449 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:53:50,449 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:50,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:50,449 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:53:50,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:53:50,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:50,450 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:50,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:50,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:53:50,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:53:50,451 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:50,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:53:50,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:50,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:53:50,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:50,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:50,455 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:50,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:53:50,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:50,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:50,458 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:53:50,458 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:53:50,458 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:53:50,459 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:53:50,459 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:53:50,462 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:50,462 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:50,462 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:50,462 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:50,462 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:50,462 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:50,462 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:50,468 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:53:50,468 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:53:50,468 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:53:50,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:53:50,469 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:53:50,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:53:50,501 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=157 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/.tmp/C/d67b46721e6f4698881fd6d66dde1c0b 2023-04-16 00:53:50,512 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/.tmp/A/c60ca2650c964955988021e770d8da8f as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/A/c60ca2650c964955988021e770d8da8f 2023-04-16 00:53:50,525 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/A/c60ca2650c964955988021e770d8da8f, entries=200, sequenceid=157, filesize=14.1 K 2023-04-16 00:53:50,525 DEBUG [MemStoreFlusher.0] regionserver.StoreScanner(961): StoreScanner already closing. There is no need to updateReaders 2023-04-16 00:53:50,528 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/.tmp/B/376f37c0b97947a1aabe342a7d2e924d as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/B/376f37c0b97947a1aabe342a7d2e924d 2023-04-16 00:53:50,536 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/B/376f37c0b97947a1aabe342a7d2e924d, entries=150, sequenceid=157, filesize=11.8 K 2023-04-16 00:53:50,539 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/.tmp/C/d67b46721e6f4698881fd6d66dde1c0b as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/C/d67b46721e6f4698881fd6d66dde1c0b 2023-04-16 00:53:50,545 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/C/d67b46721e6f4698881fd6d66dde1c0b, entries=150, sequenceid=157, filesize=11.8 K 2023-04-16 00:53:50,548 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.08 KB/130125 for c9039369968c422a52a2c7db83b430ba in 707ms, sequenceid=157, compaction requested=false 2023-04-16 00:53:50,548 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for c9039369968c422a52a2c7db83b430ba: 2023-04-16 00:53:50,559 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:53:50,560 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:53:50,567 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(9158): Flush requested on c9039369968c422a52a2c7db83b430ba 2023-04-16 00:53:50,567 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing c9039369968c422a52a2c7db83b430ba 3/3 column families, dataSize=133.78 KB heapSize=384.07 KB 2023-04-16 00:53:50,567 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c9039369968c422a52a2c7db83b430ba, store=A 2023-04-16 00:53:50,567 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 00:53:50,567 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c9039369968c422a52a2c7db83b430ba, store=B 2023-04-16 00:53:50,567 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 00:53:50,567 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c9039369968c422a52a2c7db83b430ba, store=C 2023-04-16 00:53:50,568 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 00:53:50,569 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:53:50,572 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:53:50,572 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:53:50,572 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:53:50,572 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:53:50,572 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:53:50,572 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:53:50,573 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:50,573 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:50,575 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:53:50,575 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:50,575 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:53:50,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:53:50,575 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:50,575 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:53:50,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:50,577 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:50,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:53:50,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:50,578 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:53:50,579 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:53:50,579 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:53:50,579 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:53:50,579 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:53:50,579 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:53:50,581 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:53:50,581 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:53:50,581 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. 2023-04-16 00:53:50,581 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. started... 2023-04-16 00:53:50,581 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. as already flushing 2023-04-16 00:53:50,581 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. 2023-04-16 00:53:50,581 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:53:50,581 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:53:50,581 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:50,582 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:53:50,582 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:53:50,590 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:50,590 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:50,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:50,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:53:50,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:53:50,590 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:50,590 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:53:50,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:53:50,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:53:50,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:50,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:50,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:53:50,592 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:53:50,592 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@20002929[Count = 0] remaining members to acquire global barrier 2023-04-16 00:53:50,592 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:53:50,593 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:50,594 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:50,594 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:50,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:50,594 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:53:50,594 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:53:50,594 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:53:50,594 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:50,594 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:53:50,596 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:50,596 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:53:50,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:50,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:53:50,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:53:50,596 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:53:50,596 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:53:50,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:53:50,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:53:50,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:50,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:50,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:53:50,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:50,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:50,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:53:50,599 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:53:50,599 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:53:50,599 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:53:50,599 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:53:50,599 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:53:50,599 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:53:50,601 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:50,601 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:50,601 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:50,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:50,601 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:53:50,602 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:53:50,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:53:50,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:50,602 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:50,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:50,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:53:50,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:53:50,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:53:50,608 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:50,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:50,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:53:50,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:50,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:50,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:53:50,612 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:53:50,612 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:53:50,612 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:53:50,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:53:50,613 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:50,613 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:50,613 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:50,613 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:50,613 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:50,613 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:50,613 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:50,613 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:53:50,613 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:53:50,613 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:53:50,613 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:53:50,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:53:50,614 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:53:50,614 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:53:50,614 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:53:50,621 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:53:50,621 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40044 deadline: 1681606490615, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:50,622 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:53:50,622 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40000 deadline: 1681606490616, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:50,640 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:53:50,640 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39980 deadline: 1681606490638, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:50,650 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:53:50,650 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40046 deadline: 1681606490650, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:50,655 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:53:50,655 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39990 deadline: 1681606490653, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:50,714 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:53:50,715 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:53:50,717 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:53:50,719 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:53:50,719 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:53:50,719 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:53:50,719 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:53:50,719 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:53:50,719 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:53:50,720 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:50,720 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:50,721 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:53:50,721 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:53:50,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:53:50,721 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:50,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:50,722 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:50,722 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:53:50,725 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:50,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:53:50,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:50,725 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:53:50,726 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:53:50,726 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:53:50,726 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:53:50,726 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:53:50,726 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:53:50,726 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:53:50,726 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:53:50,726 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. 2023-04-16 00:53:50,726 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. started... 2023-04-16 00:53:50,727 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. as already flushing 2023-04-16 00:53:50,727 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. 2023-04-16 00:53:50,727 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:53:50,727 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:53:50,727 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:53:50,727 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:50,727 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40000 deadline: 1681606490724, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:50,727 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:53:50,728 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:53:50,728 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:53:50,728 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40044 deadline: 1681606490724, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:50,730 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:50,730 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:50,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:50,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:53:50,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:53:50,731 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:50,731 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:53:50,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:53:50,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:53:50,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:50,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:50,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:53:50,733 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:53:50,733 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@48fedf3f[Count = 0] remaining members to acquire global barrier 2023-04-16 00:53:50,733 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:53:50,733 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:50,734 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:50,734 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:50,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:50,734 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:53:50,734 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:53:50,734 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:53:50,735 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:50,735 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:53:50,737 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:53:50,737 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:53:50,737 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:53:50,737 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:50,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:50,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:53:50,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:53:50,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:53:50,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:53:50,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:50,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:50,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:53:50,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:50,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:50,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:53:50,743 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:53:50,743 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:53:50,743 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:53:50,743 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:53:50,744 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:53:50,744 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:53:50,746 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:50,746 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:53:50,746 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:50,746 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:50,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:50,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:53:50,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:53:50,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:50,747 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:53:50,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:53:50,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:53:50,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:50,747 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:50,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:50,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:53:50,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:50,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:50,748 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:50,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:53:50,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:50,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:50,752 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:50,752 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:50,752 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:50,752 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:53:50,752 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:50,752 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:53:50,752 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:53:50,752 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:53:50,752 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:53:50,752 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:53:50,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:53:50,752 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:50,752 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:50,752 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:50,753 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:53:50,753 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:53:50,753 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:53:50,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:53:50,853 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:53:50,854 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:53:50,858 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:53:50,860 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:53:50,860 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:53:50,860 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:53:50,860 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:53:50,860 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:53:50,860 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:53:50,861 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:50,861 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:50,862 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:50,862 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:53:50,862 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:53:50,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:53:50,863 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:50,863 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:53:50,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:50,863 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:50,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:53:50,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:50,864 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:53:50,867 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:53:50,867 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:53:50,867 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:53:50,867 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:53:50,867 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:53:50,872 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:53:50,872 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:53:50,872 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. 2023-04-16 00:53:50,872 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. started... 2023-04-16 00:53:50,872 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. as already flushing 2023-04-16 00:53:50,872 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. 2023-04-16 00:53:50,872 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:53:50,872 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:53:50,872 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:50,872 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:53:50,872 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:53:50,878 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:50,878 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:50,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:50,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:53:50,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:53:50,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:53:50,879 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:50,879 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:53:50,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:53:50,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:50,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:50,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:53:50,881 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:53:50,881 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4e1ab1f4[Count = 0] remaining members to acquire global barrier 2023-04-16 00:53:50,881 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:53:50,881 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:50,882 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:50,882 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:50,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:50,882 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:53:50,882 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:53:50,882 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:50,883 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:53:50,882 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:53:50,884 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:53:50,884 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:53:50,884 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:53:50,884 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:50,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:50,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:53:50,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:53:50,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:53:50,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:53:50,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:50,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:50,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:53:50,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:50,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:50,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:53:50,891 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:53:50,891 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:53:50,891 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:53:50,891 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:53:50,891 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:53:50,891 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:53:50,894 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:50,894 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:53:50,894 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:50,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:50,894 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:50,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:50,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:53:50,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:53:50,895 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:53:50,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:53:50,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:53:50,895 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:50,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:50,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:50,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:53:50,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:50,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:50,898 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:50,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:53:50,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:50,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:50,902 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:50,902 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:53:50,902 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:50,902 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:53:50,902 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:53:50,902 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:50,902 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:53:50,902 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:53:50,902 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:53:50,902 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:50,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:53:50,902 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:50,902 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:50,902 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:50,903 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:53:50,903 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:53:50,903 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:53:50,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:53:50,932 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:53:50,933 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40044 deadline: 1681606490929, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:50,933 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:53:50,933 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40000 deadline: 1681606490930, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:51,003 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:53:51,004 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:53:51,006 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:53:51,008 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:53:51,008 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:53:51,008 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:53:51,008 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:53:51,008 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:53:51,008 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:53:51,009 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:51,009 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:51,009 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.33 KB at sequenceid=197 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/.tmp/A/5a2badaf98cb4718a64362d6e8c10d3d 2023-04-16 00:53:51,012 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:53:51,013 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:53:51,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:53:51,013 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:51,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:51,014 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:51,014 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:51,014 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:53:51,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:53:51,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:51,014 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:53:51,015 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:53:51,015 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:53:51,015 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:53:51,015 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:53:51,015 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:53:51,020 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:53:51,020 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:53:51,022 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. 2023-04-16 00:53:51,022 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. started... 2023-04-16 00:53:51,022 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. as already flushing 2023-04-16 00:53:51,022 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. 2023-04-16 00:53:51,022 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:53:51,022 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:53:51,022 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:51,022 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:53:51,022 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:53:51,024 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:51,024 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:51,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:51,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:53:51,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:53:51,025 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:51,025 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:53:51,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:53:51,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:53:51,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:51,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:51,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:53:51,034 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:53:51,034 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7a51caff[Count = 0] remaining members to acquire global barrier 2023-04-16 00:53:51,034 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:53:51,034 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:51,036 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:51,036 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:51,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:51,036 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:53:51,036 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:53:51,036 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:53:51,036 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:51,036 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:53:51,038 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:53:51,038 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:51,038 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:53:51,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:51,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:53:51,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:53:51,038 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:53:51,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:53:51,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:53:51,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:51,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:51,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:53:51,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:51,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:51,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:53:51,042 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:53:51,042 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:53:51,042 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:53:51,042 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:53:51,042 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:53:51,042 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:53:51,046 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:51,046 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:53:51,046 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:51,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:51,046 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:53:51,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:53:51,046 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:51,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:51,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:53:51,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:53:51,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:51,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:53:51,047 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:51,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:51,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:53:51,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:51,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:51,050 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:51,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:53:51,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:51,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:51,053 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:53:51,053 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:53:51,053 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:53:51,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:53:51,053 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:51,053 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:53:51,053 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:53:51,053 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:51,053 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:51,053 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:51,053 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:51,053 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:51,053 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:51,053 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:53:51,054 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:53:51,054 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:53:51,054 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:53:51,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:53:51,077 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=197 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/.tmp/B/40e5aebebbca4ac6b6b21b87079ed6e9 2023-04-16 00:53:51,127 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=197 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/.tmp/C/6a682d3996474f509ae80c126aebdc7a 2023-04-16 00:53:51,141 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/.tmp/A/5a2badaf98cb4718a64362d6e8c10d3d as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/A/5a2badaf98cb4718a64362d6e8c10d3d 2023-04-16 00:53:51,147 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:53:51,147 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39980 deadline: 1681606491142, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:51,149 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/A/5a2badaf98cb4718a64362d6e8c10d3d, entries=150, sequenceid=197, filesize=11.8 K 2023-04-16 00:53:51,152 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/.tmp/B/40e5aebebbca4ac6b6b21b87079ed6e9 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/B/40e5aebebbca4ac6b6b21b87079ed6e9 2023-04-16 00:53:51,154 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:53:51,155 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:53:51,160 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:53:51,161 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:53:51,161 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40046 deadline: 1681606491155, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:51,162 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:53:51,162 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39990 deadline: 1681606491160, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:51,163 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:53:51,163 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:53:51,164 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:53:51,164 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:53:51,164 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:53:51,164 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:53:51,164 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:51,164 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:51,165 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/B/40e5aebebbca4ac6b6b21b87079ed6e9, entries=150, sequenceid=197, filesize=11.8 K 2023-04-16 00:53:51,167 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:53:51,167 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:53:51,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:53:51,167 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:51,167 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/.tmp/C/6a682d3996474f509ae80c126aebdc7a as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/C/6a682d3996474f509ae80c126aebdc7a 2023-04-16 00:53:51,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:51,168 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:51,168 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:53:51,168 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:51,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:53:51,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:51,169 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:53:51,173 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:53:51,174 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:53:51,174 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:53:51,174 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:53:51,174 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:53:51,174 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:53:51,174 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:53:51,174 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. 2023-04-16 00:53:51,174 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. started... 2023-04-16 00:53:51,175 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. as already flushing 2023-04-16 00:53:51,175 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. 2023-04-16 00:53:51,175 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:53:51,175 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:53:51,175 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:51,175 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:53:51,175 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:53:51,176 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:51,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:51,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:53:51,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:53:51,177 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:51,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:53:51,177 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:51,177 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:53:51,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:53:51,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:51,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:51,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:53:51,181 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:53:51,181 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5bb9fdfd[Count = 0] remaining members to acquire global barrier 2023-04-16 00:53:51,181 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:53:51,181 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:51,184 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:51,185 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:51,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:51,185 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:53:51,185 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:53:51,185 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:53:51,185 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:51,185 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:53:51,186 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:53:51,186 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:51,186 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:53:51,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:51,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:53:51,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:53:51,186 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:53:51,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:53:51,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:53:51,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:51,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:51,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:53:51,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:51,189 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/C/6a682d3996474f509ae80c126aebdc7a, entries=150, sequenceid=197, filesize=11.8 K 2023-04-16 00:53:51,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:51,194 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~133.78 KB/136995, heapSize ~384.02 KB/393240, currentSize=46.96 KB/48090 for c9039369968c422a52a2c7db83b430ba in 627ms, sequenceid=197, compaction requested=true 2023-04-16 00:53:51,194 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for c9039369968c422a52a2c7db83b430ba: 2023-04-16 00:53:51,194 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 00:53:51,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:53:51,194 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-16 00:53:51,194 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 00:53:51,194 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-16 00:53:51,194 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 00:53:51,194 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:53:51,195 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:53:51,195 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:53:51,194 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:53:51,195 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:53:51,195 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:53:51,200 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36348 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-16 00:53:51,200 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 38738 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-16 00:53:51,200 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HStore(1912): c9039369968c422a52a2c7db83b430ba/A is initiating minor compaction (all files) 2023-04-16 00:53:51,200 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of c9039369968c422a52a2c7db83b430ba/A in TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. 2023-04-16 00:53:51,200 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/A/9bfdf85328814f7d9f54fd8988643218, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/A/c60ca2650c964955988021e770d8da8f, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/A/5a2badaf98cb4718a64362d6e8c10d3d] into tmpdir=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/.tmp, totalSize=37.8 K 2023-04-16 00:53:51,200 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HStore(1912): c9039369968c422a52a2c7db83b430ba/B is initiating minor compaction (all files) 2023-04-16 00:53:51,201 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HRegion(2259): Starting compaction of c9039369968c422a52a2c7db83b430ba/B in TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. 2023-04-16 00:53:51,201 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/B/b9d2f4ed4a644dcaa49ae82bb4bb1d6c, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/B/376f37c0b97947a1aabe342a7d2e924d, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/B/40e5aebebbca4ac6b6b21b87079ed6e9] into tmpdir=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/.tmp, totalSize=35.5 K 2023-04-16 00:53:51,201 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.Compactor(207): Compacting 9bfdf85328814f7d9f54fd8988643218, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=146, earliestPutTs=1681606429181 2023-04-16 00:53:51,203 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting b9d2f4ed4a644dcaa49ae82bb4bb1d6c, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=146, earliestPutTs=1681606429181 2023-04-16 00:53:51,203 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.Compactor(207): Compacting c60ca2650c964955988021e770d8da8f, keycount=200, bloomtype=ROW, size=14.1 K, encoding=NONE, compression=NONE, seqNum=157, earliestPutTs=1681606429501 2023-04-16 00:53:51,205 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.Compactor(207): Compacting 5a2badaf98cb4718a64362d6e8c10d3d, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=197, earliestPutTs=1681606430014 2023-04-16 00:53:51,205 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting 376f37c0b97947a1aabe342a7d2e924d, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=157, earliestPutTs=1681606429501 2023-04-16 00:53:51,205 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting 40e5aebebbca4ac6b6b21b87079ed6e9, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=197, earliestPutTs=1681606430014 2023-04-16 00:53:51,210 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:51,210 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:53:51,211 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:51,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:51,211 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:53:51,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:53:51,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:51,216 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:51,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:51,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:53:51,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:53:51,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:53:51,218 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:51,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:51,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:53:51,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:51,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:51,222 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:51,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:53:51,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:51,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:51,225 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:53:51,225 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:53:51,225 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:53:51,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:53:51,226 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:53:51,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:53:51,226 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:51,226 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:51,226 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:51,226 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:51,226 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:51,226 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:53:51,226 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:51,226 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:51,226 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:53:51,226 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:53:51,226 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:53:51,227 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:53:51,227 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:53:51,234 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] throttle.PressureAwareThroughputController(145): c9039369968c422a52a2c7db83b430ba#B#compaction#30 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-16 00:53:51,237 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(9158): Flush requested on c9039369968c422a52a2c7db83b430ba 2023-04-16 00:53:51,237 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing c9039369968c422a52a2c7db83b430ba 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-16 00:53:51,237 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] throttle.PressureAwareThroughputController(145): c9039369968c422a52a2c7db83b430ba#A#compaction#31 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-16 00:53:51,237 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c9039369968c422a52a2c7db83b430ba, store=A 2023-04-16 00:53:51,238 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:53:51,238 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c9039369968c422a52a2c7db83b430ba, store=B 2023-04-16 00:53:51,238 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:53:51,238 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c9039369968c422a52a2c7db83b430ba, store=C 2023-04-16 00:53:51,238 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:53:51,284 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1681606431268, max timestamp=1681606431282Num uniques 859; 2023-04-16 00:53:51,284 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1681606431268, max timestamp=1681606431282Num uniques 859; 2023-04-16 00:53:51,284 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-16 00:53:51,285 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1681606431268, max timestamp=1681606431282Num uniques 859; 2023-04-16 00:53:51,285 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-16 00:53:51,285 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-16 00:53:51,327 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:53:51,328 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:53:51,333 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:53:51,338 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:53:51,338 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:53:51,338 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:53:51,338 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:53:51,338 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:53:51,338 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:53:51,339 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:51,339 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:51,340 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:53:51,340 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:51,340 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:53:51,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:53:51,341 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:51,341 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:53:51,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:51,342 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:51,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:53:51,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:51,343 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:53:51,346 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:53:51,346 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:53:51,347 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:53:51,347 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:53:51,347 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:53:51,356 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:53:51,356 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:53:51,356 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40044 deadline: 1681606491354, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:51,356 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:53:51,356 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:53:51,357 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40000 deadline: 1681606491356, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:51,361 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. 2023-04-16 00:53:51,361 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. started... 2023-04-16 00:53:51,362 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. as already flushing 2023-04-16 00:53:51,362 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. 2023-04-16 00:53:51,362 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:53:51,362 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:53:51,362 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:51,362 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:53:51,362 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:53:51,364 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:51,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:51,364 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:51,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:53:51,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:53:51,365 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:51,365 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:53:51,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:53:51,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:53:51,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:51,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:51,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:53:51,367 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:53:51,368 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6500d692[Count = 0] remaining members to acquire global barrier 2023-04-16 00:53:51,368 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:53:51,368 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:51,373 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:51,373 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:51,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:51,373 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:53:51,373 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:53:51,373 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:53:51,373 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:51,373 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:53:51,375 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:53:51,375 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:53:51,375 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:51,375 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:53:51,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:51,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:53:51,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:53:51,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:53:51,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:53:51,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:51,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:51,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:53:51,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:51,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:51,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:53:51,379 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:53:51,380 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:53:51,380 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:53:51,380 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:53:51,380 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:53:51,380 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:53:51,381 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:51,381 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:51,381 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:51,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:51,382 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:53:51,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:51,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:53:51,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:53:51,382 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:53:51,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:53:51,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:53:51,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:51,383 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:51,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:51,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:53:51,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:51,384 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:51,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:51,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:53:51,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:51,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:51,387 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:51,387 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:51,387 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:51,387 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:53:51,387 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:53:51,387 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:53:51,387 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:51,388 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:51,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:53:51,388 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:53:51,388 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:53:51,388 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:51,388 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:51,388 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:53:51,388 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:53:51,388 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:53:51,388 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:53:51,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:53:51,388 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:53:51,463 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:53:51,463 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:53:51,464 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40044 deadline: 1681606491462, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:51,464 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40000 deadline: 1681606491462, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:51,488 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:53:51,489 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:53:51,491 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:53:51,492 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:53:51,493 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:53:51,493 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:53:51,493 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:53:51,493 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:53:51,493 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:53:51,493 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:51,493 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:51,494 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:51,495 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:53:51,495 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:53:51,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:53:51,495 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:51,495 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:53:51,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:51,495 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:51,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:53:51,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:51,496 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:53:51,496 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:53:51,496 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:53:51,496 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:53:51,496 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:53:51,496 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:53:51,496 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:53:51,496 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:53:51,497 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. 2023-04-16 00:53:51,497 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. started... 2023-04-16 00:53:51,497 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. as already flushing 2023-04-16 00:53:51,497 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. 2023-04-16 00:53:51,497 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:53:51,497 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:53:51,497 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:51,497 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:53:51,497 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:53:51,498 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:51,498 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:51,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:51,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:53:51,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:53:51,499 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:51,499 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:53:51,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:53:51,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:53:51,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:51,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:51,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:53:51,501 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:53:51,502 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7e8e4ed7[Count = 0] remaining members to acquire global barrier 2023-04-16 00:53:51,502 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:53:51,502 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:51,503 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:51,503 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:51,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:51,503 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:53:51,503 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:53:51,503 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:53:51,504 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:51,504 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:53:51,506 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:51,506 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:53:51,507 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:53:51,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:51,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:53:51,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:53:51,507 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:53:51,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:53:51,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:53:51,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:51,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:51,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:53:51,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:51,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:51,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:53:51,509 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:53:51,509 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:53:51,509 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:53:51,509 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:53:51,509 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:53:51,509 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:53:51,511 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:51,511 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:51,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:51,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:53:51,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:53:51,511 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:51,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:53:51,511 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:53:51,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:51,512 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:51,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:51,513 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:53:51,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:53:51,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:53:51,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:51,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:51,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:51,514 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:51,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:53:51,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:51,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:51,518 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:51,518 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:51,518 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:51,518 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:51,518 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:51,518 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:53:51,518 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:53:51,518 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:53:51,518 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:51,518 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:53:51,518 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:53:51,518 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:51,518 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:53:51,518 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:53:51,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:53:51,518 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:53:51,518 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:53:51,519 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:53:51,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:53:51,618 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:53:51,619 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:53:51,628 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:53:51,630 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:53:51,630 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:53:51,630 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:53:51,630 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:53:51,630 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:53:51,630 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:53:51,630 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:51,631 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:51,632 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:53:51,632 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:51,632 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:53:51,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:53:51,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:51,633 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:51,634 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:53:51,634 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:51,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:53:51,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:51,634 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:53:51,635 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:53:51,635 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:53:51,635 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:53:51,635 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:53:51,635 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:53:51,636 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:53:51,636 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:53:51,636 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. 2023-04-16 00:53:51,636 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. started... 2023-04-16 00:53:51,636 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. as already flushing 2023-04-16 00:53:51,636 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. 2023-04-16 00:53:51,636 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:53:51,636 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:53:51,636 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:51,636 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:53:51,636 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:53:51,638 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:51,638 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:51,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:51,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:53:51,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:53:51,639 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:51,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:53:51,639 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:53:51,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:53:51,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:51,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:51,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:53:51,640 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:53:51,640 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@afc65cb[Count = 0] remaining members to acquire global barrier 2023-04-16 00:53:51,640 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:53:51,641 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:51,642 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:51,642 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:51,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:51,642 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:51,642 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:53:51,642 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:53:51,642 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:53:51,642 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:53:51,643 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:51,643 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:53:51,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:51,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:53:51,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:53:51,644 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:53:51,644 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:53:51,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:53:51,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:53:51,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:51,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:51,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:53:51,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:51,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:51,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:53:51,647 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:53:51,647 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:53:51,647 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:53:51,647 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:53:51,647 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:53:51,647 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:53:51,649 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:51,649 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:53:51,649 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:51,649 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:51,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:51,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:51,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:53:51,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:53:51,649 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:51,649 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:53:51,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:53:51,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:53:51,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:51,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:51,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:53:51,651 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:51,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:51,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:51,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:53:51,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:51,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:51,654 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:53:51,654 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:51,654 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:53:51,654 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:53:51,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:53:51,655 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:53:51,655 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:53:51,655 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:53:51,654 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:51,655 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:51,655 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:51,655 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:51,655 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:51,655 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:51,655 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:53:51,655 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:53:51,676 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=208 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/.tmp/A/44badbef30794eb5842e924db40c4e1a 2023-04-16 00:53:51,676 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:53:51,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:53:51,678 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/.tmp/B/eb864ce94bb3479292ae3da5dc0d297d as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/B/eb864ce94bb3479292ae3da5dc0d297d 2023-04-16 00:53:51,679 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:53:51,683 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:53:51,683 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40044 deadline: 1681606491677, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:51,684 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:53:51,684 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40000 deadline: 1681606491683, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:51,689 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/.tmp/A/8ff8ab416d904d868be9cdfec79d02e6 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/A/8ff8ab416d904d868be9cdfec79d02e6 2023-04-16 00:53:51,708 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in c9039369968c422a52a2c7db83b430ba/B of c9039369968c422a52a2c7db83b430ba into eb864ce94bb3479292ae3da5dc0d297d(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 00:53:51,708 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in c9039369968c422a52a2c7db83b430ba/A of c9039369968c422a52a2c7db83b430ba into 8ff8ab416d904d868be9cdfec79d02e6(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 00:53:51,709 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for c9039369968c422a52a2c7db83b430ba: 2023-04-16 00:53:51,709 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba., storeName=c9039369968c422a52a2c7db83b430ba/A, priority=13, startTime=1681606431194; duration=0sec 2023-04-16 00:53:51,709 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 00:53:51,709 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-16 00:53:51,709 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for c9039369968c422a52a2c7db83b430ba: 2023-04-16 00:53:51,709 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba., storeName=c9039369968c422a52a2c7db83b430ba/B, priority=13, startTime=1681606431194; duration=0sec 2023-04-16 00:53:51,709 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 00:53:51,711 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36348 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-16 00:53:51,711 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HStore(1912): c9039369968c422a52a2c7db83b430ba/C is initiating minor compaction (all files) 2023-04-16 00:53:51,711 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of c9039369968c422a52a2c7db83b430ba/C in TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. 2023-04-16 00:53:51,711 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/C/e7ef5613df4745038a32473837327c7b, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/C/d67b46721e6f4698881fd6d66dde1c0b, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/C/6a682d3996474f509ae80c126aebdc7a] into tmpdir=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/.tmp, totalSize=35.5 K 2023-04-16 00:53:51,712 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.Compactor(207): Compacting e7ef5613df4745038a32473837327c7b, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=146, earliestPutTs=1681606429181 2023-04-16 00:53:51,712 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.Compactor(207): Compacting d67b46721e6f4698881fd6d66dde1c0b, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=157, earliestPutTs=1681606429501 2023-04-16 00:53:51,714 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.Compactor(207): Compacting 6a682d3996474f509ae80c126aebdc7a, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=197, earliestPutTs=1681606430014 2023-04-16 00:53:51,749 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] throttle.PressureAwareThroughputController(145): c9039369968c422a52a2c7db83b430ba#C#compaction#34 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-16 00:53:51,755 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:53:51,765 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:53:51,770 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:53:51,772 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:53:51,772 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:53:51,772 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:53:51,772 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:53:51,772 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:53:51,772 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:53:51,772 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:51,772 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:51,775 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:53:51,775 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:53:51,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:53:51,775 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:51,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:51,776 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:51,776 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:53:51,776 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:51,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:53:51,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:51,777 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:53:51,777 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:53:51,777 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:53:51,777 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:53:51,777 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:53:51,777 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:53:51,777 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:53:51,778 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. 2023-04-16 00:53:51,778 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. started... 2023-04-16 00:53:51,778 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. as already flushing 2023-04-16 00:53:51,778 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. 2023-04-16 00:53:51,778 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:53:51,778 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:53:51,778 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:51,778 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:53:51,778 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:53:51,780 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:51,780 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:51,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:51,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:53:51,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:53:51,780 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:51,780 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:53:51,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:53:51,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:53:51,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:51,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:51,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:53:51,783 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:53:51,783 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1e3e377c[Count = 0] remaining members to acquire global barrier 2023-04-16 00:53:51,783 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:53:51,783 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:51,785 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:51,785 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:51,785 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:53:51,788 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:51,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:51,789 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:53:51,789 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:53:51,789 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:53:51,793 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:53:51,793 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:51,793 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:53:51,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:51,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:53:51,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:53:51,793 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:53:51,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:53:51,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:53:51,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:51,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:51,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:53:51,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:51,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:51,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:53:51,797 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:53:51,797 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:53:51,797 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:53:51,797 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:53:51,797 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:53:51,797 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:53:51,799 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:51,799 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:53:51,799 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:51,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:51,799 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:51,800 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:51,800 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:53:51,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:53:51,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:51,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:53:51,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:53:51,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:51,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:53:51,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:51,801 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:51,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:53:51,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:51,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:51,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:53:51,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:51,806 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:51,806 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:51,806 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:51,806 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:53:51,806 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:51,806 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:51,806 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:51,806 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:51,806 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:53:51,806 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:53:51,806 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:53:51,806 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:53:51,806 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:53:51,808 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:53:51,808 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:53:51,808 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:53:51,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:53:51,808 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:53:51,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:53:51,906 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:53:51,907 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:53:51,910 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:53:51,911 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:53:51,911 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:53:51,911 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:53:51,911 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:53:51,911 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:53:51,911 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:53:51,912 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:51,912 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:51,915 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:53:51,915 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:51,915 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:53:51,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:53:51,915 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:51,915 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:53:51,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:51,916 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:51,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:53:51,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:51,916 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:53:51,916 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:53:51,917 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:53:51,917 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:53:51,917 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:53:51,917 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:53:51,917 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:53:51,917 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:53:51,917 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. 2023-04-16 00:53:51,917 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. started... 2023-04-16 00:53:51,917 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. as already flushing 2023-04-16 00:53:51,918 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. 2023-04-16 00:53:51,918 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:53:51,918 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:53:51,918 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:51,918 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:53:51,918 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:53:51,919 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:51,919 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:51,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:51,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:53:51,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:53:51,920 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:51,920 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:53:51,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:53:51,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:53:51,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:51,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:51,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:53:51,922 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:53:51,922 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7d69c66f[Count = 0] remaining members to acquire global barrier 2023-04-16 00:53:51,922 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:53:51,923 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:51,924 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:51,924 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:51,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:51,924 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:53:51,924 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:53:51,924 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:53:51,924 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:51,924 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:53:51,926 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:51,926 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:53:51,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:51,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:53:51,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:53:51,926 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:53:51,926 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:53:51,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:53:51,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:53:51,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:51,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:51,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:53:51,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:51,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:51,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:53:51,929 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:53:51,929 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:53:51,929 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:53:51,929 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:53:51,929 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:53:51,929 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:53:51,932 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:51,932 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:53:51,932 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:51,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:51,932 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:53:51,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:53:51,933 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:51,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:51,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:53:51,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:53:51,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:51,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:53:51,934 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:51,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:51,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:53:51,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:51,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:51,935 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:51,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:53:51,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:51,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:51,938 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:51,938 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:53:51,938 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:51,938 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:53:51,938 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:51,938 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:51,938 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:53:51,938 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:53:51,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:53:51,938 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:51,938 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:51,938 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:51,938 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:53:51,938 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:53:51,938 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:53:51,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:53:51,939 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:53:51,939 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:53:51,988 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:53:51,988 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40044 deadline: 1681606491987, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:51,990 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:53:51,991 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40000 deadline: 1681606491989, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:52,039 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:53:52,039 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:53:52,042 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:53:52,043 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:53:52,043 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:53:52,043 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:53:52,043 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:53:52,043 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:53:52,043 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:53:52,043 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:52,043 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:52,044 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:53:52,044 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:52,045 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:53:52,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:53:52,045 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:52,045 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:53:52,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:52,045 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:52,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:53:52,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:52,046 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:53:52,046 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:53:52,046 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:53:52,046 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:53:52,046 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:53:52,046 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:53:52,046 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:53:52,046 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:53:52,046 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. 2023-04-16 00:53:52,046 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. started... 2023-04-16 00:53:52,046 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. as already flushing 2023-04-16 00:53:52,047 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. 2023-04-16 00:53:52,047 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:53:52,047 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:53:52,047 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:52,047 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:53:52,047 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:53:52,048 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:52,048 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:52,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:52,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:53:52,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:53:52,048 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:52,049 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:53:52,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:53:52,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:53:52,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:52,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:52,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:53:52,050 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:53:52,050 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@711dd7e3[Count = 0] remaining members to acquire global barrier 2023-04-16 00:53:52,050 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:53:52,050 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:52,052 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:52,052 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:52,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:52,052 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:53:52,052 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:53:52,052 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:53:52,052 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:52,053 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:53:52,054 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:52,054 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:53:52,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:52,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:53:52,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:53:52,054 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:53:52,054 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:53:52,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:53:52,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:53:52,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:52,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:52,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:53:52,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:52,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:52,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:53:52,057 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:53:52,057 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:53:52,057 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:53:52,057 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:53:52,057 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:53:52,057 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:53:52,059 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:52,059 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:53:52,059 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:52,059 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:52,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:52,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:52,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:53:52,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:53:52,059 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:53:52,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:53:52,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:53:52,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:52,060 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:52,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:52,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:53:52,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:52,061 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:52,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:52,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:53:52,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:52,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:52,063 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:52,063 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:53:52,063 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:52,063 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:53:52,064 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:53:52,064 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:52,064 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:52,064 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:52,064 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:52,064 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:52,064 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:53:52,064 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:53:52,064 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:53:52,064 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:53:52,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:53:52,064 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:53:52,064 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:53:52,065 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:53:52,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:53:52,125 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=208 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/.tmp/B/545db2cd89a34c40835683ea6a86a263 2023-04-16 00:53:52,149 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:53:52,149 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39980 deadline: 1681606492148, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:52,151 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=208 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/.tmp/C/cdb2e022925e4317841f458eb47f2f92 2023-04-16 00:53:52,160 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/.tmp/A/44badbef30794eb5842e924db40c4e1a as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/A/44badbef30794eb5842e924db40c4e1a 2023-04-16 00:53:52,164 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:53:52,165 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:53:52,167 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:53:52,168 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:53:52,168 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39990 deadline: 1681606492163, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:52,169 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:53:52,169 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40046 deadline: 1681606492164, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:52,172 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:53:52,173 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:53:52,173 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:53:52,173 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:53:52,173 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:53:52,173 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:53:52,178 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:52,178 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:52,179 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/A/44badbef30794eb5842e924db40c4e1a, entries=200, sequenceid=208, filesize=14.1 K 2023-04-16 00:53:52,180 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:52,180 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:53:52,180 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:53:52,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:53:52,183 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:52,183 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:53:52,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:52,184 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:52,184 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/.tmp/B/545db2cd89a34c40835683ea6a86a263 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/B/545db2cd89a34c40835683ea6a86a263 2023-04-16 00:53:52,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:53:52,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:52,184 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:53:52,187 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/.tmp/C/2564d901e187492689e4c014f1735db1 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/C/2564d901e187492689e4c014f1735db1 2023-04-16 00:53:52,187 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:53:52,187 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:53:52,187 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:53:52,187 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:53:52,187 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:53:52,188 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:53:52,188 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:53:52,188 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. 2023-04-16 00:53:52,188 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. started... 2023-04-16 00:53:52,188 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. as already flushing 2023-04-16 00:53:52,188 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. 2023-04-16 00:53:52,188 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:53:52,188 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:53:52,188 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:52,188 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:53:52,188 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:53:52,190 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:52,190 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:52,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:52,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:53:52,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:53:52,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:53:52,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:53:52,191 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:52,191 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:53:52,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:52,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:52,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:53:52,193 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:53:52,193 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@18a63636[Count = 0] remaining members to acquire global barrier 2023-04-16 00:53:52,193 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:53:52,194 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:52,194 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/B/545db2cd89a34c40835683ea6a86a263, entries=150, sequenceid=208, filesize=11.8 K 2023-04-16 00:53:52,195 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/.tmp/C/cdb2e022925e4317841f458eb47f2f92 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/C/cdb2e022925e4317841f458eb47f2f92 2023-04-16 00:53:52,196 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:52,196 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:52,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:52,196 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:53:52,196 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:53:52,196 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:53:52,196 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:52,196 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:53:52,197 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:52,197 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:53:52,197 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:53:52,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:52,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:53:52,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:53:52,198 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:53:52,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:53:52,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:53:52,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:52,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:52,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:53:52,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:52,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:52,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:53:52,201 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:53:52,201 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:53:52,201 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:53:52,201 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:53:52,201 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:53:52,201 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:53:52,203 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/C/cdb2e022925e4317841f458eb47f2f92, entries=150, sequenceid=208, filesize=11.8 K 2023-04-16 00:53:52,204 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=134.18 KB/137400 for c9039369968c422a52a2c7db83b430ba in 967ms, sequenceid=208, compaction requested=false 2023-04-16 00:53:52,204 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for c9039369968c422a52a2c7db83b430ba: 2023-04-16 00:53:52,205 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:52,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:52,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:53:52,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:53:52,205 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:52,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:53:52,205 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:52,205 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:53:52,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:52,206 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:52,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:52,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:53:52,206 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:53:52,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:53:52,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:52,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:52,207 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:52,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:53:52,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:52,208 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in c9039369968c422a52a2c7db83b430ba/C of c9039369968c422a52a2c7db83b430ba into 2564d901e187492689e4c014f1735db1(size=12.1 K), total size for store is 23.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 00:53:52,208 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for c9039369968c422a52a2c7db83b430ba: 2023-04-16 00:53:52,208 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba., storeName=c9039369968c422a52a2c7db83b430ba/C, priority=13, startTime=1681606431194; duration=0sec 2023-04-16 00:53:52,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:52,208 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 00:53:52,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:52,214 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:53:52,214 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:53:52,214 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:52,214 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:52,214 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:52,214 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:52,214 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:53:52,214 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:53:52,214 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:52,214 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:53:52,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:53:52,214 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:53:52,214 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:53:52,214 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:53:52,214 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:52,214 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:52,214 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:53:52,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:53:52,314 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:53:52,315 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:53:52,317 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:53:52,319 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:53:52,319 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:53:52,319 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:53:52,319 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:53:52,319 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:53:52,319 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:53:52,319 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:52,319 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:52,321 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:53:52,321 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:52,321 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:53:52,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:53:52,321 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:52,321 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:53:52,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:52,322 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:52,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:53:52,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:52,322 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:53:52,323 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:53:52,323 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:53:52,323 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:53:52,323 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:53:52,323 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:53:52,323 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:53:52,323 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:53:52,323 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. 2023-04-16 00:53:52,324 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. started... 2023-04-16 00:53:52,324 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing c9039369968c422a52a2c7db83b430ba 3/3 column families, dataSize=134.18 KB heapSize=370.92 KB 2023-04-16 00:53:52,324 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c9039369968c422a52a2c7db83b430ba, store=A 2023-04-16 00:53:52,324 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 00:53:52,324 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c9039369968c422a52a2c7db83b430ba, store=B 2023-04-16 00:53:52,324 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 00:53:52,324 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c9039369968c422a52a2c7db83b430ba, store=C 2023-04-16 00:53:52,324 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 00:53:52,343 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=245 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/.tmp/A/d1429223ed4040008faa38b9495a7a4f 2023-04-16 00:53:52,497 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. as already flushing 2023-04-16 00:53:52,497 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(9158): Flush requested on c9039369968c422a52a2c7db83b430ba 2023-04-16 00:53:52,517 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:53:52,517 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40000 deadline: 1681606492511, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:52,522 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:53:52,522 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40044 deadline: 1681606492517, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:52,620 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:53:52,621 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40000 deadline: 1681606492620, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:52,625 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:53:52,625 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40044 deadline: 1681606492624, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:52,761 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=245 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/.tmp/B/66dcf25f2dec4a07aaa142f3e50680b3 2023-04-16 00:53:52,776 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=245 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/.tmp/C/1d73dbf886d449af8949899e62596d47 2023-04-16 00:53:52,782 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/.tmp/A/d1429223ed4040008faa38b9495a7a4f as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/A/d1429223ed4040008faa38b9495a7a4f 2023-04-16 00:53:52,791 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/A/d1429223ed4040008faa38b9495a7a4f, entries=150, sequenceid=245, filesize=11.8 K 2023-04-16 00:53:52,792 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/.tmp/B/66dcf25f2dec4a07aaa142f3e50680b3 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/B/66dcf25f2dec4a07aaa142f3e50680b3 2023-04-16 00:53:52,799 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/B/66dcf25f2dec4a07aaa142f3e50680b3, entries=150, sequenceid=245, filesize=11.8 K 2023-04-16 00:53:52,800 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/.tmp/C/1d73dbf886d449af8949899e62596d47 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/C/1d73dbf886d449af8949899e62596d47 2023-04-16 00:53:52,812 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/C/1d73dbf886d449af8949899e62596d47, entries=150, sequenceid=245, filesize=11.8 K 2023-04-16 00:53:52,813 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~370.88 KB/379776, currentSize=53.67 KB/54960 for c9039369968c422a52a2c7db83b430ba in 489ms, sequenceid=245, compaction requested=true 2023-04-16 00:53:52,813 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for c9039369968c422a52a2c7db83b430ba: 2023-04-16 00:53:52,813 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. 2023-04-16 00:53:52,813 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:53:52,814 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:53:52,814 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:52,814 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:53:52,814 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:53:52,816 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:52,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:52,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:53:52,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:53:52,816 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:52,817 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:52,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:53:52,817 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:53:52,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:53:52,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:52,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:52,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:53:52,818 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:53:52,818 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@77cc6789[Count = 0] remaining members to acquire global barrier 2023-04-16 00:53:52,818 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:53:52,819 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:52,820 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:52,820 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:52,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:52,820 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:53:52,820 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:53:52,820 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:53:52,820 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:52,820 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:53:52,822 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:53:52,822 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:52,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:52,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:53:52,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:53:52,822 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:53:52,823 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:53:52,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:53:52,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:53:52,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:52,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:52,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:53:52,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:52,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:52,825 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(9158): Flush requested on c9039369968c422a52a2c7db83b430ba 2023-04-16 00:53:52,825 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing c9039369968c422a52a2c7db83b430ba 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-16 00:53:52,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:53:52,826 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:53:52,826 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c9039369968c422a52a2c7db83b430ba, store=A 2023-04-16 00:53:52,826 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:53:52,826 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:53:52,826 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:53:52,826 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:53:52,826 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c9039369968c422a52a2c7db83b430ba, store=B 2023-04-16 00:53:52,826 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:53:52,826 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:53:52,826 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:53:52,826 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c9039369968c422a52a2c7db83b430ba, store=C 2023-04-16 00:53:52,826 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:53:52,828 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:52,828 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:53:52,828 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:52,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:52,828 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:52,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:52,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:53:52,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:53:52,829 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:53:52,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:53:52,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:52,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:53:52,830 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:52,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:52,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:53:52,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:52,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:52,832 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:52,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:53:52,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:52,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:52,835 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:53:52,835 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:53:52,835 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:52,835 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:52,835 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:53:52,835 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:53:52,835 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:53:52,835 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:53:52,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:53:52,836 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:53:52,835 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:52,836 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:52,836 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:52,836 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:52,836 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:52,836 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:53:52,836 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:53:52,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:53:52,848 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=257 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/.tmp/A/b086ffa0bbae4374b740da259db862af 2023-04-16 00:53:52,871 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=257 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/.tmp/B/b296aedb2f3e4ede859bcdc7d27ee6a0 2023-04-16 00:53:52,874 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:53:52,874 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40044 deadline: 1681606492871, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:52,878 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:53:52,879 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40000 deadline: 1681606492874, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:52,889 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=257 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/.tmp/C/08db334e63124a019778864bf984d119 2023-04-16 00:53:52,894 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/.tmp/A/b086ffa0bbae4374b740da259db862af as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/A/b086ffa0bbae4374b740da259db862af 2023-04-16 00:53:52,900 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/A/b086ffa0bbae4374b740da259db862af, entries=150, sequenceid=257, filesize=11.8 K 2023-04-16 00:53:52,901 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/.tmp/B/b296aedb2f3e4ede859bcdc7d27ee6a0 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/B/b296aedb2f3e4ede859bcdc7d27ee6a0 2023-04-16 00:53:52,907 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/B/b296aedb2f3e4ede859bcdc7d27ee6a0, entries=150, sequenceid=257, filesize=11.8 K 2023-04-16 00:53:52,909 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/.tmp/C/08db334e63124a019778864bf984d119 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/C/08db334e63124a019778864bf984d119 2023-04-16 00:53:52,917 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/C/08db334e63124a019778864bf984d119, entries=150, sequenceid=257, filesize=11.8 K 2023-04-16 00:53:52,918 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for c9039369968c422a52a2c7db83b430ba in 93ms, sequenceid=257, compaction requested=true 2023-04-16 00:53:52,919 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for c9039369968c422a52a2c7db83b430ba: 2023-04-16 00:53:52,919 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 00:53:52,919 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-16 00:53:52,919 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-16 00:53:52,919 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 00:53:52,919 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 00:53:52,921 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 50926 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-16 00:53:52,921 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HStore(1912): c9039369968c422a52a2c7db83b430ba/A is initiating minor compaction (all files) 2023-04-16 00:53:52,921 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of c9039369968c422a52a2c7db83b430ba/A in TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. 2023-04-16 00:53:52,921 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/A/8ff8ab416d904d868be9cdfec79d02e6, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/A/44badbef30794eb5842e924db40c4e1a, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/A/d1429223ed4040008faa38b9495a7a4f, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/A/b086ffa0bbae4374b740da259db862af] into tmpdir=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/.tmp, totalSize=49.7 K 2023-04-16 00:53:52,922 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48536 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-16 00:53:52,922 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.Compactor(207): Compacting 8ff8ab416d904d868be9cdfec79d02e6, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=197, earliestPutTs=1681606430014 2023-04-16 00:53:52,922 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HStore(1912): c9039369968c422a52a2c7db83b430ba/B is initiating minor compaction (all files) 2023-04-16 00:53:52,923 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HRegion(2259): Starting compaction of c9039369968c422a52a2c7db83b430ba/B in TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. 2023-04-16 00:53:52,923 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/B/eb864ce94bb3479292ae3da5dc0d297d, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/B/545db2cd89a34c40835683ea6a86a263, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/B/66dcf25f2dec4a07aaa142f3e50680b3, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/B/b296aedb2f3e4ede859bcdc7d27ee6a0] into tmpdir=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/.tmp, totalSize=47.4 K 2023-04-16 00:53:52,924 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting eb864ce94bb3479292ae3da5dc0d297d, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=197, earliestPutTs=1681606430014 2023-04-16 00:53:52,924 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.Compactor(207): Compacting 44badbef30794eb5842e924db40c4e1a, keycount=200, bloomtype=ROW, size=14.1 K, encoding=NONE, compression=NONE, seqNum=208, earliestPutTs=1681606430581 2023-04-16 00:53:52,925 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.Compactor(207): Compacting d1429223ed4040008faa38b9495a7a4f, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=245, earliestPutTs=1681606431290 2023-04-16 00:53:52,925 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.Compactor(207): Compacting b086ffa0bbae4374b740da259db862af, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=257, earliestPutTs=1681606432510 2023-04-16 00:53:52,927 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting 545db2cd89a34c40835683ea6a86a263, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=208, earliestPutTs=1681606430581 2023-04-16 00:53:52,929 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting 66dcf25f2dec4a07aaa142f3e50680b3, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=245, earliestPutTs=1681606431290 2023-04-16 00:53:52,930 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting b296aedb2f3e4ede859bcdc7d27ee6a0, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=257, earliestPutTs=1681606432510 2023-04-16 00:53:52,936 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:53:52,937 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:53:52,939 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:53:52,944 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:53:52,944 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:53:52,944 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] throttle.PressureAwareThroughputController(145): c9039369968c422a52a2c7db83b430ba#A#compaction#42 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-16 00:53:52,944 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:53:52,944 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:53:52,944 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:53:52,944 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:53:52,945 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:52,945 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:52,947 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:52,947 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:53:52,947 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:53:52,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:53:52,948 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:52,948 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:53:52,948 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] throttle.PressureAwareThroughputController(145): c9039369968c422a52a2c7db83b430ba#B#compaction#43 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-16 00:53:52,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:52,948 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:52,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:53:52,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:52,950 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:53:52,959 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:53:52,959 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:53:52,959 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:53:52,959 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:53:52,960 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:53:52,961 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:53:52,961 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:53:52,962 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. 2023-04-16 00:53:52,962 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. started... 2023-04-16 00:53:52,962 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing c9039369968c422a52a2c7db83b430ba 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-16 00:53:52,973 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c9039369968c422a52a2c7db83b430ba, store=A 2023-04-16 00:53:52,973 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:53:52,973 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c9039369968c422a52a2c7db83b430ba, store=B 2023-04-16 00:53:52,973 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:53:52,973 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c9039369968c422a52a2c7db83b430ba, store=C 2023-04-16 00:53:52,973 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:53:52,984 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(9158): Flush requested on c9039369968c422a52a2c7db83b430ba 2023-04-16 00:53:52,984 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. as already flushing 2023-04-16 00:53:52,998 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=277 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/.tmp/A/1905893bd8cf465dbb2f31993b65478d 2023-04-16 00:53:53,001 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:53:53,001 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40000 deadline: 1681606492999, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:53,003 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:53:53,004 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40044 deadline: 1681606493001, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:53,106 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:53:53,106 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40000 deadline: 1681606493103, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:53,107 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:53:53,107 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40044 deadline: 1681606493105, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:53,309 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:53:53,309 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40000 deadline: 1681606493309, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:53,310 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:53:53,310 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40044 deadline: 1681606493309, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:53,376 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/.tmp/A/4ce5db29f17d44ed8dfa68525748f796 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/A/4ce5db29f17d44ed8dfa68525748f796 2023-04-16 00:53:53,385 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in c9039369968c422a52a2c7db83b430ba/A of c9039369968c422a52a2c7db83b430ba into 4ce5db29f17d44ed8dfa68525748f796(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 00:53:53,385 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for c9039369968c422a52a2c7db83b430ba: 2023-04-16 00:53:53,385 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba., storeName=c9039369968c422a52a2c7db83b430ba/A, priority=12, startTime=1681606432919; duration=0sec 2023-04-16 00:53:53,385 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 00:53:53,386 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-16 00:53:53,387 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48536 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-16 00:53:53,387 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HStore(1912): c9039369968c422a52a2c7db83b430ba/C is initiating minor compaction (all files) 2023-04-16 00:53:53,387 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of c9039369968c422a52a2c7db83b430ba/C in TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. 2023-04-16 00:53:53,387 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/C/2564d901e187492689e4c014f1735db1, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/C/cdb2e022925e4317841f458eb47f2f92, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/C/1d73dbf886d449af8949899e62596d47, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/C/08db334e63124a019778864bf984d119] into tmpdir=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/.tmp, totalSize=47.4 K 2023-04-16 00:53:53,388 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.Compactor(207): Compacting 2564d901e187492689e4c014f1735db1, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=197, earliestPutTs=1681606430014 2023-04-16 00:53:53,389 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.Compactor(207): Compacting cdb2e022925e4317841f458eb47f2f92, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=208, earliestPutTs=1681606430581 2023-04-16 00:53:53,389 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.Compactor(207): Compacting 1d73dbf886d449af8949899e62596d47, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=245, earliestPutTs=1681606431290 2023-04-16 00:53:53,399 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.Compactor(207): Compacting 08db334e63124a019778864bf984d119, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=257, earliestPutTs=1681606432510 2023-04-16 00:53:53,399 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/.tmp/B/43f42b9fb9624279850656e3c66e092d as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/B/43f42b9fb9624279850656e3c66e092d 2023-04-16 00:53:53,407 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in c9039369968c422a52a2c7db83b430ba/B of c9039369968c422a52a2c7db83b430ba into 43f42b9fb9624279850656e3c66e092d(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 00:53:53,407 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for c9039369968c422a52a2c7db83b430ba: 2023-04-16 00:53:53,407 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba., storeName=c9039369968c422a52a2c7db83b430ba/B, priority=12, startTime=1681606432919; duration=0sec 2023-04-16 00:53:53,407 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 00:53:53,412 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] throttle.PressureAwareThroughputController(145): c9039369968c422a52a2c7db83b430ba#C#compaction#46 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-16 00:53:53,425 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=277 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/.tmp/B/0a9e1b474e824a118b82cfe404453cbb 2023-04-16 00:53:53,613 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:53:53,613 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40000 deadline: 1681606493612, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:53,613 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:53:53,613 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40044 deadline: 1681606493613, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:53,828 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/.tmp/C/c1e8f2ef7e1f4ec0895c7a1b6b1bebd7 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/C/c1e8f2ef7e1f4ec0895c7a1b6b1bebd7 2023-04-16 00:53:53,834 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in c9039369968c422a52a2c7db83b430ba/C of c9039369968c422a52a2c7db83b430ba into c1e8f2ef7e1f4ec0895c7a1b6b1bebd7(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 00:53:53,834 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for c9039369968c422a52a2c7db83b430ba: 2023-04-16 00:53:53,834 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba., storeName=c9039369968c422a52a2c7db83b430ba/C, priority=12, startTime=1681606432919; duration=0sec 2023-04-16 00:53:53,834 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 00:53:53,870 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=277 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/.tmp/C/ed3ec1d5ae5e4da5a297370347b5db2b 2023-04-16 00:53:53,876 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/.tmp/A/1905893bd8cf465dbb2f31993b65478d as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/A/1905893bd8cf465dbb2f31993b65478d 2023-04-16 00:53:53,882 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/A/1905893bd8cf465dbb2f31993b65478d, entries=150, sequenceid=277, filesize=11.9 K 2023-04-16 00:53:53,883 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/.tmp/B/0a9e1b474e824a118b82cfe404453cbb as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/B/0a9e1b474e824a118b82cfe404453cbb 2023-04-16 00:53:53,888 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/B/0a9e1b474e824a118b82cfe404453cbb, entries=150, sequenceid=277, filesize=11.9 K 2023-04-16 00:53:53,889 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/.tmp/C/ed3ec1d5ae5e4da5a297370347b5db2b as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/C/ed3ec1d5ae5e4da5a297370347b5db2b 2023-04-16 00:53:53,893 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/C/ed3ec1d5ae5e4da5a297370347b5db2b, entries=150, sequenceid=277, filesize=11.9 K 2023-04-16 00:53:53,894 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for c9039369968c422a52a2c7db83b430ba in 932ms, sequenceid=277, compaction requested=false 2023-04-16 00:53:53,895 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for c9039369968c422a52a2c7db83b430ba: 2023-04-16 00:53:53,895 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. 2023-04-16 00:53:53,895 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:53:53,896 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:53:53,896 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:53,896 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:53:53,896 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:53:53,901 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:53,901 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:53,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:53,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:53:53,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:53:53,901 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:53,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:53:53,902 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:53:53,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:53:53,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:53,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:53,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:53:53,903 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:53:53,903 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@44259da1[Count = 0] remaining members to acquire global barrier 2023-04-16 00:53:53,903 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:53:53,903 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:53,905 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:53,905 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:53,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:53,905 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:53:53,905 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:53:53,905 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:53,905 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:53:53,905 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:53:53,908 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:53:53,908 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:53:53,908 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:53:53,908 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:53,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:53,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:53:53,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:53:53,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:53:53,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:53:53,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:53,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:53,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:53:53,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:53,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:53,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:53:53,911 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:53:53,911 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:53:53,911 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:53:53,911 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:53:53,911 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:53:53,911 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:53:53,913 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:53,913 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:53,913 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:53,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:53,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:53:53,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:53:53,913 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:53:53,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:53,914 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:53:53,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:53:53,914 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:53,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:53:53,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:53,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:53,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:53:53,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:53,915 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:53,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:53,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:53:53,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:53,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:53,918 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:53:53,918 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:53:53,918 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:53:53,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:53:53,918 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:53:53,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:53:53,921 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:53,921 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:53,921 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:53,921 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:53,921 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:53:53,921 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:53:53,921 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:53:53,921 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:53,921 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:53,921 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:53,922 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:53:53,922 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:53:54,022 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:53:54,023 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:53:54,025 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:53:54,027 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:53:54,027 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:53:54,027 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:53:54,027 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:53:54,027 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:53:54,027 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:53:54,036 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:54,036 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:54,037 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:53:54,037 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:54,037 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:53:54,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:53:54,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:54,038 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:54,038 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:53:54,038 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:54,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:53:54,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:54,039 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:53:54,039 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:53:54,039 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:53:54,039 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:53:54,039 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:53:54,039 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:53:54,040 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:53:54,040 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:53:54,040 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. 2023-04-16 00:53:54,040 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. started... 2023-04-16 00:53:54,040 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing c9039369968c422a52a2c7db83b430ba 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-16 00:53:54,041 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c9039369968c422a52a2c7db83b430ba, store=A 2023-04-16 00:53:54,041 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:53:54,041 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c9039369968c422a52a2c7db83b430ba, store=B 2023-04-16 00:53:54,041 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:53:54,041 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c9039369968c422a52a2c7db83b430ba, store=C 2023-04-16 00:53:54,041 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:53:54,079 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=292 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/.tmp/A/466fae954ec74a619f64c7c59333c1d5 2023-04-16 00:53:54,105 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=292 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/.tmp/B/0598e1459d21477abb94ec7af539950c 2023-04-16 00:53:54,125 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(9158): Flush requested on c9039369968c422a52a2c7db83b430ba 2023-04-16 00:53:54,125 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. as already flushing 2023-04-16 00:53:54,135 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=292 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/.tmp/C/048858e9a71e4b60a0689027894b6451 2023-04-16 00:53:54,149 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/.tmp/A/466fae954ec74a619f64c7c59333c1d5 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/A/466fae954ec74a619f64c7c59333c1d5 2023-04-16 00:53:54,155 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/A/466fae954ec74a619f64c7c59333c1d5, entries=150, sequenceid=292, filesize=11.9 K 2023-04-16 00:53:54,156 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/.tmp/B/0598e1459d21477abb94ec7af539950c as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/B/0598e1459d21477abb94ec7af539950c 2023-04-16 00:53:54,162 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/B/0598e1459d21477abb94ec7af539950c, entries=150, sequenceid=292, filesize=11.9 K 2023-04-16 00:53:54,163 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:53:54,166 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681606434130, max timestamp=1681606434162Num uniques 859; 2023-04-16 00:53:54,166 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681606434130, max timestamp=1681606434162Num uniques 859; 2023-04-16 00:53:54,166 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-16 00:53:54,166 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-16 00:53:54,166 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39980 deadline: 1681606494155, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:54,168 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/.tmp/C/048858e9a71e4b60a0689027894b6451 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/C/048858e9a71e4b60a0689027894b6451 2023-04-16 00:53:54,169 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681606434130, max timestamp=1681606434162Num uniques 859; 2023-04-16 00:53:54,169 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-16 00:53:54,177 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/C/048858e9a71e4b60a0689027894b6451, entries=150, sequenceid=292, filesize=11.9 K 2023-04-16 00:53:54,180 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=31.31 KB/32060 for c9039369968c422a52a2c7db83b430ba in 140ms, sequenceid=292, compaction requested=true 2023-04-16 00:53:54,180 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for c9039369968c422a52a2c7db83b430ba: 2023-04-16 00:53:54,180 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. 2023-04-16 00:53:54,180 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:53:54,180 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:53:54,180 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:54,182 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:53:54,182 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:53:54,185 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:54,185 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:54,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:54,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:53:54,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:53:54,186 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:54,186 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:53:54,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:53:54,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:53:54,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:54,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:54,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:53:54,188 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(9158): Flush requested on c9039369968c422a52a2c7db83b430ba 2023-04-16 00:53:54,188 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:53:54,188 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing c9039369968c422a52a2c7db83b430ba 3/3 column families, dataSize=58.14 KB heapSize=156.08 KB 2023-04-16 00:53:54,188 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:53:54,189 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:54,188 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5b18f809[Count = 0] remaining members to acquire global barrier 2023-04-16 00:53:54,189 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c9039369968c422a52a2c7db83b430ba, store=A 2023-04-16 00:53:54,189 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 00:53:54,189 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c9039369968c422a52a2c7db83b430ba, store=B 2023-04-16 00:53:54,189 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 00:53:54,189 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c9039369968c422a52a2c7db83b430ba, store=C 2023-04-16 00:53:54,189 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 00:53:54,190 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:54,190 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:54,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:54,190 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:53:54,190 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:53:54,190 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:54,190 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:53:54,190 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:53:54,192 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:53:54,193 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:53:54,193 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:53:54,193 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:54,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:54,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:53:54,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:53:54,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:53:54,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:53:54,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:54,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:54,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:53:54,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:54,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:54,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:53:54,202 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:53:54,202 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:53:54,202 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:53:54,202 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:53:54,202 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:53:54,202 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:53:54,204 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:54,204 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:54,204 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:54,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:54,204 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:53:54,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:54,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:53:54,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:53:54,204 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:53:54,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:53:54,205 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:54,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:53:54,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:54,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:54,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:53:54,206 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=317 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/.tmp/A/84134e01213642f18323474e33fe7f2d 2023-04-16 00:53:54,207 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:54,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:54,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:54,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:53:54,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:54,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:54,214 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:54,214 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:54,214 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:54,214 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:54,214 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:54,214 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:54,214 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:53:54,214 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:53:54,214 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:53:54,214 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:53:54,214 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:54,214 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:53:54,214 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:53:54,214 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:53:54,214 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:53:54,214 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:53:54,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:53:54,215 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:53:54,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:53:54,232 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681606434219, max timestamp=1681606434229Num uniques 859; 2023-04-16 00:53:54,233 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-16 00:53:54,234 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681606434219, max timestamp=1681606434229Num uniques 859; 2023-04-16 00:53:54,234 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-16 00:53:54,234 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681606434219, max timestamp=1681606434229Num uniques 859; 2023-04-16 00:53:54,234 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-16 00:53:54,290 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:53:54,290 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39990 deadline: 1681606494280, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:54,291 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:53:54,291 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40044 deadline: 1681606494282, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:54,291 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:53:54,291 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40000 deadline: 1681606494283, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:54,292 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:53:54,292 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39980 deadline: 1681606494288, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:54,296 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:53:54,296 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40046 deadline: 1681606494290, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:54,315 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:53:54,315 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:53:54,319 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:53:54,331 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:53:54,331 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:53:54,331 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:53:54,331 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:53:54,331 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:53:54,331 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:53:54,332 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:54,332 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:54,334 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:54,335 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:53:54,335 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:53:54,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:53:54,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:54,335 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:54,335 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:53:54,336 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:54,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:53:54,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:54,337 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:53:54,338 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:53:54,338 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:53:54,338 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:53:54,338 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:53:54,338 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:53:54,340 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:53:54,340 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:53:54,340 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. 2023-04-16 00:53:54,340 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. started... 2023-04-16 00:53:54,340 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. as already flushing 2023-04-16 00:53:54,340 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. 2023-04-16 00:53:54,340 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:53:54,340 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:53:54,340 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:54,340 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:53:54,340 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:53:54,342 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:54,342 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:54,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:54,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:53:54,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:53:54,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:53:54,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:53:54,344 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:54,344 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:53:54,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:54,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:54,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:53:54,348 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:53:54,348 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@722672da[Count = 0] remaining members to acquire global barrier 2023-04-16 00:53:54,348 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:53:54,348 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:54,350 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:54,350 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:54,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:54,351 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:53:54,351 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:53:54,351 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:53:54,351 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:54,351 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:53:54,352 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:54,352 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:53:54,352 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:53:54,353 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:53:54,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:54,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:53:54,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:53:54,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:53:54,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:53:54,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:54,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:54,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:53:54,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:54,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:54,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:53:54,360 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:53:54,360 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:53:54,360 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:53:54,360 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:53:54,360 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:53:54,360 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:53:54,366 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:54,366 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:53:54,366 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:54,366 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:54,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:54,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:53:54,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:53:54,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:54,367 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:53:54,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:53:54,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:54,368 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:54,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:53:54,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:54,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:53:54,371 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:54,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:54,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:54,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:53:54,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:54,374 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:53:54,375 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:53:54,375 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:53:54,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:53:54,375 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:54,375 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:54,375 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:54,375 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:54,375 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:53:54,375 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:53:54,375 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:53:54,375 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:54,375 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:54,375 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:54,375 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:53:54,375 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:53:54,375 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:53:54,376 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:53:54,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:53:54,393 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:53:54,393 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39990 deadline: 1681606494392, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:54,393 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:53:54,394 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40044 deadline: 1681606494392, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:54,394 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:53:54,394 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40000 deadline: 1681606494393, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:54,394 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:53:54,395 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39980 deadline: 1681606494393, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:54,399 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:53:54,399 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40046 deadline: 1681606494397, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:54,484 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:53:54,486 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:53:54,496 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:53:54,497 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:53:54,498 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:53:54,498 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:53:54,498 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:53:54,498 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:53:54,498 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:53:54,498 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:54,498 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:54,499 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:53:54,499 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:54,499 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:53:54,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:53:54,500 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:54,500 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:53:54,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:54,500 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:54,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:53:54,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:54,501 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:53:54,501 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:53:54,501 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:53:54,501 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:53:54,501 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:53:54,501 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:53:54,501 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:53:54,501 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. 2023-04-16 00:53:54,502 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. started... 2023-04-16 00:53:54,502 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. as already flushing 2023-04-16 00:53:54,502 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. 2023-04-16 00:53:54,502 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:53:54,502 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:53:54,502 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:54,502 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:53:54,502 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:53:54,503 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:54,503 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:54,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:54,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:53:54,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:53:54,504 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:54,504 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:53:54,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:53:54,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:53:54,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:54,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:54,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:53:54,506 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:53:54,506 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6642b338[Count = 0] remaining members to acquire global barrier 2023-04-16 00:53:54,506 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:53:54,506 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:54,508 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:54,508 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:54,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:54,508 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:53:54,508 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:53:54,508 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:53:54,509 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:54,509 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:53:54,510 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:54,510 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:53:54,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:54,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:53:54,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:53:54,510 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:53:54,510 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:53:54,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:53:54,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:53:54,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:54,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:54,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:53:54,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:54,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:54,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:53:54,513 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:53:54,513 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:53:54,514 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:53:54,514 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:53:54,514 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:53:54,514 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:53:54,517 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:54,517 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:53:54,517 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:54,517 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:54,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:54,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:53:54,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:53:54,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:54,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:53:54,518 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:53:54,518 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:54,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:53:54,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:54,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:54,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:53:54,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:54,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:54,529 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:54,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:53:54,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:54,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:54,531 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:54,532 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:54,532 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:54,532 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:54,532 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:54,532 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:53:54,532 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:53:54,532 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:53:54,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:53:54,532 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:53:54,532 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:53:54,532 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:53:54,532 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:54,532 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:54,532 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:53:54,532 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:53:54,533 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:53:54,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:53:54,595 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:53:54,595 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39990 deadline: 1681606494595, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:54,600 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:53:54,600 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40000 deadline: 1681606494597, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:54,600 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:53:54,600 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40044 deadline: 1681606494597, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:54,601 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:53:54,602 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39980 deadline: 1681606494597, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:54,604 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:53:54,604 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40046 deadline: 1681606494601, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:54,633 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:53:54,633 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:53:54,635 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:53:54,636 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:53:54,636 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:53:54,637 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:53:54,637 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:53:54,637 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:53:54,637 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:53:54,637 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:54,637 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:54,639 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:53:54,639 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:54,639 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:53:54,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:53:54,640 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:54,640 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:53:54,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:54,640 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:54,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:53:54,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:54,645 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:53:54,645 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:53:54,645 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:53:54,645 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:53:54,645 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:53:54,645 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:53:54,646 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:53:54,646 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:53:54,646 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. 2023-04-16 00:53:54,646 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. started... 2023-04-16 00:53:54,646 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. as already flushing 2023-04-16 00:53:54,646 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. 2023-04-16 00:53:54,646 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:53:54,646 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:53:54,646 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:54,646 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:53:54,646 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:53:54,648 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:54,648 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:54,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:54,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:53:54,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:53:54,648 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:54,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:53:54,648 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:53:54,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:53:54,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:54,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:54,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:53:54,650 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:53:54,650 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@ee2e2c4[Count = 0] remaining members to acquire global barrier 2023-04-16 00:53:54,650 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:53:54,650 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:54,651 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:54,651 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:54,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:54,651 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:53:54,651 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:53:54,651 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:53:54,651 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:54,651 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:53:54,653 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:53:54,653 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:53:54,653 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:53:54,653 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:54,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:54,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:53:54,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:53:54,662 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=317 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/.tmp/B/06ffbabf500c4a3cb0393f5878f775d1 2023-04-16 00:53:54,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:53:54,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:53:54,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:54,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:54,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:53:54,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:54,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:54,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:53:54,672 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:53:54,672 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:53:54,672 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:53:54,672 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:53:54,672 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:53:54,672 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:53:54,686 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:54,686 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:54,686 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:53:54,686 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:54,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:54,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:54,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:53:54,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:53:54,686 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:53:54,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:53:54,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:53:54,687 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:54,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:54,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:54,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:53:54,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:54,690 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:54,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:54,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:53:54,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:54,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:54,692 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:54,692 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:53:54,692 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:54,693 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:53:54,693 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:53:54,693 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:54,693 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:54,693 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:54,693 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:54,693 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:54,693 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:53:54,693 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:53:54,692 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:53:54,693 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:53:54,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:53:54,693 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:53:54,693 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:53:54,694 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:53:54,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:53:54,715 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=317 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/.tmp/C/22ae222c4480457d84e9bef4e411489e 2023-04-16 00:53:54,724 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/.tmp/A/84134e01213642f18323474e33fe7f2d as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/A/84134e01213642f18323474e33fe7f2d 2023-04-16 00:53:54,736 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/A/84134e01213642f18323474e33fe7f2d, entries=250, sequenceid=317, filesize=16.7 K 2023-04-16 00:53:54,738 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/.tmp/B/06ffbabf500c4a3cb0393f5878f775d1 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/B/06ffbabf500c4a3cb0393f5878f775d1 2023-04-16 00:53:54,745 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/B/06ffbabf500c4a3cb0393f5878f775d1, entries=150, sequenceid=317, filesize=11.9 K 2023-04-16 00:53:54,747 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/.tmp/C/22ae222c4480457d84e9bef4e411489e as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/C/22ae222c4480457d84e9bef4e411489e 2023-04-16 00:53:54,755 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/C/22ae222c4480457d84e9bef4e411489e, entries=150, sequenceid=317, filesize=11.9 K 2023-04-16 00:53:54,756 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~58.14 KB/59540, heapSize ~156.03 KB/159776, currentSize=127.47 KB/130530 for c9039369968c422a52a2c7db83b430ba in 568ms, sequenceid=317, compaction requested=true 2023-04-16 00:53:54,756 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for c9039369968c422a52a2c7db83b430ba: 2023-04-16 00:53:54,756 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-16 00:53:54,757 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 00:53:54,757 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 00:53:54,757 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-16 00:53:54,757 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 00:53:54,758 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 54002 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-16 00:53:54,758 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HStore(1912): c9039369968c422a52a2c7db83b430ba/A is initiating minor compaction (all files) 2023-04-16 00:53:54,758 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of c9039369968c422a52a2c7db83b430ba/A in TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. 2023-04-16 00:53:54,758 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/A/4ce5db29f17d44ed8dfa68525748f796, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/A/1905893bd8cf465dbb2f31993b65478d, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/A/466fae954ec74a619f64c7c59333c1d5, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/A/84134e01213642f18323474e33fe7f2d] into tmpdir=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/.tmp, totalSize=52.7 K 2023-04-16 00:53:54,759 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49122 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-16 00:53:54,759 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HStore(1912): c9039369968c422a52a2c7db83b430ba/B is initiating minor compaction (all files) 2023-04-16 00:53:54,759 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HRegion(2259): Starting compaction of c9039369968c422a52a2c7db83b430ba/B in TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. 2023-04-16 00:53:54,759 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/B/43f42b9fb9624279850656e3c66e092d, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/B/0a9e1b474e824a118b82cfe404453cbb, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/B/0598e1459d21477abb94ec7af539950c, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/B/06ffbabf500c4a3cb0393f5878f775d1] into tmpdir=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/.tmp, totalSize=48.0 K 2023-04-16 00:53:54,760 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.Compactor(207): Compacting 4ce5db29f17d44ed8dfa68525748f796, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=257, earliestPutTs=1681606432510 2023-04-16 00:53:54,762 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting 43f42b9fb9624279850656e3c66e092d, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=257, earliestPutTs=1681606432510 2023-04-16 00:53:54,763 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.Compactor(207): Compacting 1905893bd8cf465dbb2f31993b65478d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=277, earliestPutTs=1681606432853 2023-04-16 00:53:54,763 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting 0a9e1b474e824a118b82cfe404453cbb, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=277, earliestPutTs=1681606432853 2023-04-16 00:53:54,764 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting 0598e1459d21477abb94ec7af539950c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=292, earliestPutTs=1681606432993 2023-04-16 00:53:54,764 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.Compactor(207): Compacting 466fae954ec74a619f64c7c59333c1d5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=292, earliestPutTs=1681606432993 2023-04-16 00:53:54,764 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting 06ffbabf500c4a3cb0393f5878f775d1, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=317, earliestPutTs=1681606434161 2023-04-16 00:53:54,765 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.Compactor(207): Compacting 84134e01213642f18323474e33fe7f2d, keycount=250, bloomtype=ROW, size=16.7 K, encoding=NONE, compression=NONE, seqNum=317, earliestPutTs=1681606434130 2023-04-16 00:53:54,782 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] throttle.PressureAwareThroughputController(145): c9039369968c422a52a2c7db83b430ba#B#compaction#54 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-16 00:53:54,782 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] throttle.PressureAwareThroughputController(145): c9039369968c422a52a2c7db83b430ba#A#compaction#55 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-16 00:53:54,793 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:53:54,794 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:53:54,802 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:53:54,808 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:53:54,808 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:53:54,808 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:53:54,808 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:53:54,808 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:53:54,808 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:53:54,809 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:54,809 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:54,813 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:53:54,813 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:53:54,813 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:54,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:53:54,815 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:54,815 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:53:54,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:54,816 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:54,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:53:54,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:54,817 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:53:54,817 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:53:54,817 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:53:54,817 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:53:54,817 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:53:54,817 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:53:54,817 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:53:54,818 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:53:54,818 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. 2023-04-16 00:53:54,818 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. started... 2023-04-16 00:53:54,819 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing c9039369968c422a52a2c7db83b430ba 3/3 column families, dataSize=127.47 KB heapSize=365.06 KB 2023-04-16 00:53:54,820 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c9039369968c422a52a2c7db83b430ba, store=A 2023-04-16 00:53:54,820 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 00:53:54,820 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c9039369968c422a52a2c7db83b430ba, store=B 2023-04-16 00:53:54,820 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 00:53:54,820 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c9039369968c422a52a2c7db83b430ba, store=C 2023-04-16 00:53:54,820 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 00:53:54,823 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/.tmp/B/93b265976d5e4ceeb91c4225461306dc as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/B/93b265976d5e4ceeb91c4225461306dc 2023-04-16 00:53:54,828 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/.tmp/A/02a8166c300b499992611a4e874179f4 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/A/02a8166c300b499992611a4e874179f4 2023-04-16 00:53:54,834 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in c9039369968c422a52a2c7db83b430ba/B of c9039369968c422a52a2c7db83b430ba into 93b265976d5e4ceeb91c4225461306dc(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 00:53:54,834 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for c9039369968c422a52a2c7db83b430ba: 2023-04-16 00:53:54,834 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba., storeName=c9039369968c422a52a2c7db83b430ba/B, priority=12, startTime=1681606434757; duration=0sec 2023-04-16 00:53:54,834 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 00:53:54,834 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-16 00:53:54,837 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49122 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-16 00:53:54,837 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HStore(1912): c9039369968c422a52a2c7db83b430ba/C is initiating minor compaction (all files) 2023-04-16 00:53:54,837 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HRegion(2259): Starting compaction of c9039369968c422a52a2c7db83b430ba/C in TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. 2023-04-16 00:53:54,837 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/C/c1e8f2ef7e1f4ec0895c7a1b6b1bebd7, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/C/ed3ec1d5ae5e4da5a297370347b5db2b, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/C/048858e9a71e4b60a0689027894b6451, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/C/22ae222c4480457d84e9bef4e411489e] into tmpdir=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/.tmp, totalSize=48.0 K 2023-04-16 00:53:54,838 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting c1e8f2ef7e1f4ec0895c7a1b6b1bebd7, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=257, earliestPutTs=1681606432510 2023-04-16 00:53:54,838 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting ed3ec1d5ae5e4da5a297370347b5db2b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=277, earliestPutTs=1681606432853 2023-04-16 00:53:54,839 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting 048858e9a71e4b60a0689027894b6451, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=292, earliestPutTs=1681606432993 2023-04-16 00:53:54,841 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting 22ae222c4480457d84e9bef4e411489e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=317, earliestPutTs=1681606434161 2023-04-16 00:53:54,860 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] throttle.PressureAwareThroughputController(145): c9039369968c422a52a2c7db83b430ba#C#compaction#56 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-16 00:53:54,862 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in c9039369968c422a52a2c7db83b430ba/A of c9039369968c422a52a2c7db83b430ba into 02a8166c300b499992611a4e874179f4(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 00:53:54,877 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for c9039369968c422a52a2c7db83b430ba: 2023-04-16 00:53:54,877 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba., storeName=c9039369968c422a52a2c7db83b430ba/A, priority=12, startTime=1681606434756; duration=0sec 2023-04-16 00:53:54,877 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 00:53:54,896 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/.tmp/C/e1a2b1285d9c41df841a226d149eaf57 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/C/e1a2b1285d9c41df841a226d149eaf57 2023-04-16 00:53:54,899 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(9158): Flush requested on c9039369968c422a52a2c7db83b430ba 2023-04-16 00:53:54,899 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. as already flushing 2023-04-16 00:53:54,902 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in c9039369968c422a52a2c7db83b430ba/C of c9039369968c422a52a2c7db83b430ba into e1a2b1285d9c41df841a226d149eaf57(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 00:53:54,902 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for c9039369968c422a52a2c7db83b430ba: 2023-04-16 00:53:54,902 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba., storeName=c9039369968c422a52a2c7db83b430ba/C, priority=12, startTime=1681606434757; duration=0sec 2023-04-16 00:53:54,903 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 00:53:54,913 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:53:54,914 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39980 deadline: 1681606494910, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:54,914 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:53:54,914 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39990 deadline: 1681606494911, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:54,918 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:53:54,918 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40044 deadline: 1681606494912, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:54,918 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:53:54,918 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:53:54,918 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40046 deadline: 1681606494913, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:54,918 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40000 deadline: 1681606494914, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:55,016 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:53:55,016 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:53:55,016 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39980 deadline: 1681606495015, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:55,017 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39990 deadline: 1681606495015, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:55,021 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:53:55,021 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 175 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40044 deadline: 1681606495019, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:55,021 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:53:55,021 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 159 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40000 deadline: 1681606495019, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:55,222 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:53:55,222 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39990 deadline: 1681606495219, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:55,227 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:53:55,228 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 177 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40044 deadline: 1681606495223, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:55,228 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:53:55,228 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 161 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40000 deadline: 1681606495223, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:55,228 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:53:55,228 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39980 deadline: 1681606495224, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:55,296 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=353 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/.tmp/A/1abf1d29f50d4662b0d87093ad802574 2023-04-16 00:53:55,342 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=353 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/.tmp/B/e8cb49583cd948629f43a90a88961c17 2023-04-16 00:53:55,366 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=353 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/.tmp/C/4f0d2eafb2174fa495279f45b84c02d5 2023-04-16 00:53:55,375 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/.tmp/A/1abf1d29f50d4662b0d87093ad802574 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/A/1abf1d29f50d4662b0d87093ad802574 2023-04-16 00:53:55,382 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/A/1abf1d29f50d4662b0d87093ad802574, entries=150, sequenceid=353, filesize=11.9 K 2023-04-16 00:53:55,383 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/.tmp/B/e8cb49583cd948629f43a90a88961c17 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/B/e8cb49583cd948629f43a90a88961c17 2023-04-16 00:53:55,398 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/B/e8cb49583cd948629f43a90a88961c17, entries=150, sequenceid=353, filesize=11.9 K 2023-04-16 00:53:55,400 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/.tmp/C/4f0d2eafb2174fa495279f45b84c02d5 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/C/4f0d2eafb2174fa495279f45b84c02d5 2023-04-16 00:53:55,406 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/C/4f0d2eafb2174fa495279f45b84c02d5, entries=150, sequenceid=353, filesize=11.9 K 2023-04-16 00:53:55,407 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~365.02 KB/373776, currentSize=53.67 KB/54960 for c9039369968c422a52a2c7db83b430ba in 588ms, sequenceid=353, compaction requested=false 2023-04-16 00:53:55,407 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for c9039369968c422a52a2c7db83b430ba: 2023-04-16 00:53:55,407 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. 2023-04-16 00:53:55,408 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:53:55,411 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:53:55,411 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:55,411 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:53:55,411 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:53:55,418 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:55,418 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:55,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:55,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:53:55,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:53:55,419 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:55,419 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:53:55,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:53:55,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:53:55,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:55,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:55,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:53:55,422 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:53:55,422 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@12268ba1[Count = 0] remaining members to acquire global barrier 2023-04-16 00:53:55,422 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:53:55,422 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:55,423 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:55,423 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:55,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:55,423 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:53:55,423 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:53:55,423 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:53:55,424 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:55,424 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:53:55,424 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(9158): Flush requested on c9039369968c422a52a2c7db83b430ba 2023-04-16 00:53:55,424 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing c9039369968c422a52a2c7db83b430ba 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-16 00:53:55,425 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c9039369968c422a52a2c7db83b430ba, store=A 2023-04-16 00:53:55,425 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:53:55,425 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:53:55,425 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c9039369968c422a52a2c7db83b430ba, store=B 2023-04-16 00:53:55,425 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:55,425 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:53:55,425 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:53:55,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:55,425 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c9039369968c422a52a2c7db83b430ba, store=C 2023-04-16 00:53:55,425 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:53:55,425 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:53:55,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:53:55,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:53:55,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:53:55,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:53:55,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:55,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:55,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:53:55,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:55,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:55,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:53:55,430 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:53:55,430 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:53:55,430 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:53:55,430 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:53:55,430 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:53:55,430 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:53:55,436 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:55,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:55,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:53:55,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:53:55,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:53:55,437 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:55,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:55,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:53:55,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:55,438 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:55,438 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:53:55,438 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:55,438 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:55,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:55,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:55,439 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:53:55,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:53:55,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:53:55,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:55,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:55,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:55,442 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:53:55,442 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:53:55,442 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:53:55,442 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:53:55,443 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:53:55,442 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:53:55,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:53:55,442 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:53:55,443 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:53:55,443 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:53:55,443 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:55,443 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:55,443 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:55,443 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:55,443 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:55,443 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:55,443 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:55,443 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:53:55,446 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:53:55,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:53:55,453 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=368 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/.tmp/A/ebc9cf8536f84732a94acbf91ee93a31 2023-04-16 00:53:55,497 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:53:55,498 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40046 deadline: 1681606495496, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:55,524 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:53:55,524 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39990 deadline: 1681606495523, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:55,531 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:53:55,531 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40000 deadline: 1681606495531, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:55,532 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:53:55,532 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:53:55,532 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 179 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40044 deadline: 1681606495531, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:55,532 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39980 deadline: 1681606495531, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:55,543 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:53:55,544 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:53:55,547 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:53:55,548 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:53:55,548 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:53:55,548 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:53:55,549 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:53:55,549 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:53:55,549 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:53:55,549 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:55,549 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:55,550 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:55,550 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:53:55,550 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:53:55,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:53:55,551 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:55,551 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:53:55,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:55,552 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:55,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:53:55,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:55,560 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:53:55,561 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:53:55,561 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:53:55,561 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:53:55,561 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:53:55,562 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:53:55,562 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:53:55,562 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:53:55,562 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. 2023-04-16 00:53:55,562 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. started... 2023-04-16 00:53:55,562 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. as already flushing 2023-04-16 00:53:55,562 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. 2023-04-16 00:53:55,562 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:53:55,562 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:53:55,562 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:55,562 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:53:55,562 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:53:55,567 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:55,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:55,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:53:55,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:53:55,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:53:55,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:53:55,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:55,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:55,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:53:55,570 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:53:55,570 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@53579b9d[Count = 0] remaining members to acquire global barrier 2023-04-16 00:53:55,570 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:53:55,570 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:55,572 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:55,572 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(162): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:55,572 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:55,572 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:53:55,572 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:53:55,572 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:53:55,572 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:53:55,573 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:55,573 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:53:55,574 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:53:55,574 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:53:55,574 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:55,575 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:53:55,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:55,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:53:55,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:53:55,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:53:55,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:53:55,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:55,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:55,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:53:55,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:55,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:55,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:53:55,580 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:53:55,580 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:53:55,580 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:53:55,580 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:53:55,580 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:53:55,580 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:53:55,586 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:55,586 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:55,586 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:55,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:55,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:53:55,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:53:55,586 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:53:55,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:55,592 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:53:55,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:53:55,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:53:55,592 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:55,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:55,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:55,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:53:55,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:55,594 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:55,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:55,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:53:55,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:55,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:55,596 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:53:55,597 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:55,597 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:53:55,597 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:53:55,597 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:55,597 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:53:55,597 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:55,597 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:53:55,597 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:53:55,597 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:53:55,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:53:55,597 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:55,597 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:55,597 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:53:55,597 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:53:55,597 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:55,598 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:55,598 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:55,598 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:53:55,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:53:55,602 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:53:55,602 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40046 deadline: 1681606495599, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:55,698 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:53:55,698 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:53:55,701 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:53:55,702 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:53:55,702 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:53:55,703 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:53:55,703 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:53:55,703 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:53:55,703 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:53:55,703 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:55,703 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:55,704 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:55,705 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:53:55,705 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:53:55,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:53:55,705 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:55,705 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:53:55,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:55,706 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:55,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:53:55,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:55,708 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:53:55,708 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:53:55,708 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:53:55,708 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:53:55,708 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:53:55,708 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:53:55,708 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:53:55,709 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:53:55,709 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. 2023-04-16 00:53:55,709 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. started... 2023-04-16 00:53:55,709 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. as already flushing 2023-04-16 00:53:55,709 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. 2023-04-16 00:53:55,709 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:53:55,709 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:53:55,709 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:55,709 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:53:55,709 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:53:55,711 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:55,711 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:55,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:55,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:53:55,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:53:55,713 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:55,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:53:55,713 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:53:55,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:53:55,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:55,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:55,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:53:55,716 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:53:55,716 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@20ff29bc[Count = 0] remaining members to acquire global barrier 2023-04-16 00:53:55,716 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:53:55,716 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:55,718 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:55,718 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:55,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:55,718 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:55,718 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:53:55,718 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:53:55,718 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:53:55,718 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:53:55,720 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:53:55,720 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:55,720 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:53:55,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:55,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:53:55,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:53:55,720 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:53:55,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:53:55,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:53:55,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:55,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:55,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:53:55,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:55,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:55,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:53:55,724 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:53:55,724 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:53:55,724 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:53:55,724 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:53:55,724 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:53:55,724 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:53:55,725 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:55,726 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:55,726 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:55,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:55,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:53:55,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:53:55,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:55,726 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:53:55,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:53:55,726 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:53:55,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:53:55,726 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:55,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:55,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:55,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:53:55,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:55,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:55,728 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:55,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:53:55,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:55,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:55,731 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:53:55,731 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:53:55,731 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:53:55,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:53:55,732 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:55,732 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:55,732 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:55,732 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:55,732 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:55,732 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:53:55,732 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:53:55,732 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:53:55,732 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:55,732 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:55,732 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:53:55,732 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:53:55,733 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:53:55,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:53:55,808 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:53:55,809 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40046 deadline: 1681606495804, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:55,832 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:53:55,834 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:53:55,837 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:53:55,838 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:53:55,838 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:53:55,838 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:53:55,838 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:53:55,839 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:53:55,839 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:53:55,839 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:55,839 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:55,840 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:53:55,840 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:55,840 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:53:55,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:53:55,841 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:55,841 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:53:55,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:55,841 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:55,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:53:55,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:55,842 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:53:55,842 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:53:55,842 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:53:55,842 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:53:55,842 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:53:55,842 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:53:55,843 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:53:55,843 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:53:55,843 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. 2023-04-16 00:53:55,843 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. started... 2023-04-16 00:53:55,843 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. as already flushing 2023-04-16 00:53:55,843 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. 2023-04-16 00:53:55,843 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:53:55,843 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:53:55,843 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:55,843 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:53:55,843 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:53:55,845 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:55,845 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:55,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:55,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:53:55,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:53:55,845 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:55,845 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:53:55,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:53:55,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:53:55,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:55,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:55,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:53:55,847 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:53:55,847 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@39120ba5[Count = 0] remaining members to acquire global barrier 2023-04-16 00:53:55,847 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:53:55,847 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:55,852 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:55,852 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:55,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:55,852 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:53:55,852 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:53:55,852 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:53:55,852 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:55,852 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:53:55,854 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:53:55,854 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:55,854 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:53:55,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:55,854 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:53:55,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:53:55,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:53:55,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:53:55,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:53:55,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:55,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:55,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:53:55,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:55,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:55,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:53:55,857 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:53:55,857 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:53:55,857 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:53:55,857 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:53:55,857 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:53:55,857 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:53:55,858 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:55,858 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:53:55,858 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:55,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:55,858 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:55,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:55,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:53:55,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:53:55,859 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:53:55,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:53:55,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:53:55,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:55,859 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:55,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:55,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:53:55,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:55,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:55,860 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:55,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:53:55,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:55,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:55,864 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:55,864 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:53:55,864 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:55,864 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:55,864 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:55,864 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:55,864 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:55,864 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:55,864 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:53:55,864 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:53:55,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:53:55,864 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:53:55,864 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:53:55,864 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:53:55,865 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:53:55,865 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:53:55,865 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:53:55,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:53:55,901 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=368 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/.tmp/B/2be077d7177e46a69ecdb89d78f53876 2023-04-16 00:53:55,965 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:53:55,965 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:53:55,967 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:53:55,968 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:53:55,968 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:53:55,968 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:53:55,968 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:53:55,969 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:53:55,969 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:53:55,969 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:55,969 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:55,971 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:55,971 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:53:55,972 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:53:55,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:53:55,972 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:55,972 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:53:55,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:55,972 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:55,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:53:55,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:55,973 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:53:55,973 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:53:55,973 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:53:55,973 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:53:55,973 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:53:55,973 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:53:55,973 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:53:55,973 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:53:55,973 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. 2023-04-16 00:53:55,973 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. started... 2023-04-16 00:53:55,973 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. as already flushing 2023-04-16 00:53:55,974 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. 2023-04-16 00:53:55,974 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:53:55,974 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:53:55,974 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:55,974 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:53:55,974 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:53:55,975 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:55,975 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:55,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:55,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:53:55,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:53:55,975 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:55,975 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:53:55,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:53:55,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:53:55,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:55,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:55,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:53:55,977 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:53:55,977 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1f4fe4a5[Count = 0] remaining members to acquire global barrier 2023-04-16 00:53:55,977 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:53:55,977 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:55,978 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:55,978 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:55,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:55,978 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:53:55,978 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:53:55,978 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:53:55,978 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:55,978 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:53:55,979 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:53:55,979 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:53:55,980 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:53:55,980 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:55,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:55,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:53:55,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:53:55,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:53:55,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:53:55,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:55,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:55,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:53:55,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:55,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:55,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:53:55,982 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:53:55,982 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:53:55,982 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:53:55,982 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:53:55,982 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:53:55,982 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:53:55,985 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:55,985 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:55,985 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:53:55,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:55,985 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:55,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:55,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:53:55,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:53:55,985 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:53:55,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:53:55,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:53:55,985 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:55,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:55,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:55,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:53:55,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:55,987 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:55,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:55,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:53:55,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:55,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:55,989 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:55,989 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:53:55,989 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:55,990 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:53:55,990 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:53:55,990 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:53:55,990 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:53:55,990 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:53:55,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:53:55,990 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:55,990 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:53:55,990 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:55,990 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:55,990 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:55,990 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:55,990 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:53:55,990 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:53:55,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:53:55,990 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:53:56,028 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:53:56,028 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39990 deadline: 1681606496027, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:56,037 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:53:56,038 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 181 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40044 deadline: 1681606496034, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:56,038 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:53:56,038 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40000 deadline: 1681606496035, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:56,038 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:53:56,038 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39980 deadline: 1681606496035, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:56,091 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:53:56,091 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:53:56,093 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:53:56,094 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:53:56,094 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:53:56,094 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:53:56,094 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:53:56,094 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:53:56,094 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:53:56,095 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:56,095 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:56,096 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:53:56,096 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:56,096 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:53:56,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:53:56,096 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:56,096 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:53:56,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:56,097 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:56,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:53:56,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:56,097 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:53:56,097 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:53:56,098 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:53:56,098 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:53:56,098 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:53:56,098 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:53:56,098 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:53:56,098 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:53:56,098 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. 2023-04-16 00:53:56,098 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. started... 2023-04-16 00:53:56,098 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. as already flushing 2023-04-16 00:53:56,098 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. 2023-04-16 00:53:56,098 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:53:56,098 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:53:56,098 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:56,098 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:53:56,098 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:53:56,100 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:56,100 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:56,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:56,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:53:56,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:53:56,100 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:56,100 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:53:56,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:53:56,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:53:56,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:56,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:56,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:53:56,102 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:53:56,102 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@70ab0fa2[Count = 0] remaining members to acquire global barrier 2023-04-16 00:53:56,102 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:53:56,102 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:56,105 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:56,105 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:56,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:56,105 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:53:56,105 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:53:56,105 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:53:56,105 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:56,105 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:53:56,106 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:53:56,106 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:56,106 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:53:56,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:56,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:53:56,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:53:56,106 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:53:56,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:53:56,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:53:56,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:56,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:56,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:53:56,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:56,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:56,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:53:56,110 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:53:56,110 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:53:56,110 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:53:56,110 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:53:56,110 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:53:56,110 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:53:56,112 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:56,112 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:56,112 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:56,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:56,112 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:53:56,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:56,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:53:56,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:53:56,112 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:53:56,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:53:56,113 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:56,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:53:56,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:56,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:56,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:53:56,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:56,114 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:56,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:56,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:53:56,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:56,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:56,116 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:53:56,116 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:56,116 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:56,116 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:56,116 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40046 deadline: 1681606496112, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:56,116 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:53:56,116 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:53:56,116 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:56,117 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:56,116 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:53:56,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:53:56,117 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:53:56,117 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:56,117 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:53:56,117 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:56,117 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:53:56,117 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:53:56,117 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:53:56,117 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:53:56,117 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:53:56,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:53:56,217 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:53:56,218 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:53:56,220 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:53:56,221 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:53:56,221 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:53:56,221 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:53:56,221 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:53:56,221 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:53:56,221 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:53:56,222 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:56,222 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:56,224 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:53:56,224 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:56,224 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:53:56,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:53:56,224 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:56,224 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:53:56,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:56,225 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:56,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:53:56,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:56,225 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:53:56,226 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:53:56,226 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:53:56,226 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:53:56,226 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:53:56,226 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:53:56,226 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:53:56,226 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:53:56,226 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. 2023-04-16 00:53:56,226 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. started... 2023-04-16 00:53:56,226 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. as already flushing 2023-04-16 00:53:56,226 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. 2023-04-16 00:53:56,226 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:53:56,226 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:53:56,226 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:56,227 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:53:56,227 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:53:56,228 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:56,228 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:56,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:56,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:53:56,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:53:56,228 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:56,229 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:53:56,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:53:56,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:53:56,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:56,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:56,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:53:56,230 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:53:56,230 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@29406ac2[Count = 0] remaining members to acquire global barrier 2023-04-16 00:53:56,230 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:53:56,230 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:56,231 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:56,231 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:56,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:56,231 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:53:56,231 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:53:56,231 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:53:56,231 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:56,231 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:53:56,237 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:56,237 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:53:56,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:56,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:53:56,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:53:56,237 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:53:56,237 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:53:56,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:53:56,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:53:56,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:56,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:56,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:53:56,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:56,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:56,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:53:56,239 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:53:56,239 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:53:56,239 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:53:56,239 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:53:56,239 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:53:56,239 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:53:56,241 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:56,241 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:56,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:56,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:53:56,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:53:56,241 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:56,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:56,241 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:53:56,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:53:56,241 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:56,241 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:53:56,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:53:56,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:56,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:56,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:53:56,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:56,242 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:56,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:56,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:53:56,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:56,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:56,244 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:56,244 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:56,244 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:56,245 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:56,245 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:56,245 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:53:56,245 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:53:56,245 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:53:56,245 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:56,245 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:56,245 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:53:56,245 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:53:56,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:53:56,245 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:53:56,245 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:53:56,245 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:53:56,245 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:53:56,245 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:53:56,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:53:56,316 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=368 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/.tmp/C/68fe0c007b144c26af83d9c2ef895eec 2023-04-16 00:53:56,323 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/.tmp/A/ebc9cf8536f84732a94acbf91ee93a31 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/A/ebc9cf8536f84732a94acbf91ee93a31 2023-04-16 00:53:56,333 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/A/ebc9cf8536f84732a94acbf91ee93a31, entries=150, sequenceid=368, filesize=11.9 K 2023-04-16 00:53:56,334 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/.tmp/B/2be077d7177e46a69ecdb89d78f53876 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/B/2be077d7177e46a69ecdb89d78f53876 2023-04-16 00:53:56,346 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/B/2be077d7177e46a69ecdb89d78f53876, entries=150, sequenceid=368, filesize=11.9 K 2023-04-16 00:53:56,346 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:53:56,347 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/.tmp/C/68fe0c007b144c26af83d9c2ef895eec as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/C/68fe0c007b144c26af83d9c2ef895eec 2023-04-16 00:53:56,347 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:53:56,351 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:53:56,353 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:53:56,353 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:53:56,353 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:53:56,353 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:53:56,353 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:53:56,353 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:53:56,354 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:56,354 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:56,355 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/C/68fe0c007b144c26af83d9c2ef895eec, entries=150, sequenceid=368, filesize=11.9 K 2023-04-16 00:53:56,355 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:53:56,355 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:56,355 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:53:56,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:53:56,356 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for c9039369968c422a52a2c7db83b430ba in 932ms, sequenceid=368, compaction requested=true 2023-04-16 00:53:56,356 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for c9039369968c422a52a2c7db83b430ba: 2023-04-16 00:53:56,357 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-16 00:53:56,357 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 00:53:56,357 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 00:53:56,357 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-16 00:53:56,357 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-16 00:53:56,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:56,358 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:56,358 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:53:56,358 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:56,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:53:56,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:56,359 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:53:56,359 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:53:56,359 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:53:56,359 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:53:56,359 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:53:56,359 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:53:56,361 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37172 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-16 00:53:56,361 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HStore(1912): c9039369968c422a52a2c7db83b430ba/A is initiating minor compaction (all files) 2023-04-16 00:53:56,361 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of c9039369968c422a52a2c7db83b430ba/A in TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. 2023-04-16 00:53:56,361 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/A/02a8166c300b499992611a4e874179f4, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/A/1abf1d29f50d4662b0d87093ad802574, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/A/ebc9cf8536f84732a94acbf91ee93a31] into tmpdir=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/.tmp, totalSize=36.3 K 2023-04-16 00:53:56,362 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.Compactor(207): Compacting 02a8166c300b499992611a4e874179f4, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=317, earliestPutTs=1681606434161 2023-04-16 00:53:56,363 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.Compactor(207): Compacting 1abf1d29f50d4662b0d87093ad802574, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=353, earliestPutTs=1681606434280 2023-04-16 00:53:56,363 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37172 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-16 00:53:56,368 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:53:56,374 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HStore(1912): c9039369968c422a52a2c7db83b430ba/B is initiating minor compaction (all files) 2023-04-16 00:53:56,365 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.Compactor(207): Compacting ebc9cf8536f84732a94acbf91ee93a31, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=368, earliestPutTs=1681606434911 2023-04-16 00:53:56,375 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HRegion(2259): Starting compaction of c9039369968c422a52a2c7db83b430ba/B in TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. 2023-04-16 00:53:56,375 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/B/93b265976d5e4ceeb91c4225461306dc, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/B/e8cb49583cd948629f43a90a88961c17, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/B/2be077d7177e46a69ecdb89d78f53876] into tmpdir=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/.tmp, totalSize=36.3 K 2023-04-16 00:53:56,374 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:53:56,376 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting 93b265976d5e4ceeb91c4225461306dc, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=317, earliestPutTs=1681606434161 2023-04-16 00:53:56,376 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting e8cb49583cd948629f43a90a88961c17, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=353, earliestPutTs=1681606434280 2023-04-16 00:53:56,374 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. 2023-04-16 00:53:56,377 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. started... 2023-04-16 00:53:56,377 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing c9039369968c422a52a2c7db83b430ba 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-16 00:53:56,378 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting 2be077d7177e46a69ecdb89d78f53876, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=368, earliestPutTs=1681606434911 2023-04-16 00:53:56,380 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c9039369968c422a52a2c7db83b430ba, store=A 2023-04-16 00:53:56,380 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:53:56,380 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c9039369968c422a52a2c7db83b430ba, store=B 2023-04-16 00:53:56,380 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:53:56,380 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c9039369968c422a52a2c7db83b430ba, store=C 2023-04-16 00:53:56,380 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:53:56,385 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] throttle.PressureAwareThroughputController(145): c9039369968c422a52a2c7db83b430ba#A#compaction#63 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-16 00:53:56,389 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] throttle.PressureAwareThroughputController(145): c9039369968c422a52a2c7db83b430ba#B#compaction#64 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-16 00:53:56,413 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=388 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/.tmp/A/5306221b1e9e44fb8abfe2f45f1bb09d 2023-04-16 00:53:56,623 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(9158): Flush requested on c9039369968c422a52a2c7db83b430ba 2023-04-16 00:53:56,623 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. as already flushing 2023-04-16 00:53:56,649 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:53:56,650 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40046 deadline: 1681606496646, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:56,753 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:53:56,753 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40046 deadline: 1681606496751, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:56,818 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/.tmp/B/9a5fce57a03746c4910f4096856433c3 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/B/9a5fce57a03746c4910f4096856433c3 2023-04-16 00:53:56,819 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/.tmp/A/956450698f4741c890069edfacfe3aed as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/A/956450698f4741c890069edfacfe3aed 2023-04-16 00:53:56,826 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=388 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/.tmp/B/a5d40af55a374c5cb2763ce0ba9afb42 2023-04-16 00:53:56,835 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in c9039369968c422a52a2c7db83b430ba/A of c9039369968c422a52a2c7db83b430ba into 956450698f4741c890069edfacfe3aed(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 00:53:56,835 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for c9039369968c422a52a2c7db83b430ba: 2023-04-16 00:53:56,835 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba., storeName=c9039369968c422a52a2c7db83b430ba/A, priority=13, startTime=1681606436357; duration=0sec 2023-04-16 00:53:56,835 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 00:53:56,835 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-16 00:53:56,837 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37172 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-16 00:53:56,838 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HStore(1912): c9039369968c422a52a2c7db83b430ba/C is initiating minor compaction (all files) 2023-04-16 00:53:56,838 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of c9039369968c422a52a2c7db83b430ba/C in TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. 2023-04-16 00:53:56,838 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/C/e1a2b1285d9c41df841a226d149eaf57, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/C/4f0d2eafb2174fa495279f45b84c02d5, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/C/68fe0c007b144c26af83d9c2ef895eec] into tmpdir=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/.tmp, totalSize=36.3 K 2023-04-16 00:53:56,838 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.Compactor(207): Compacting e1a2b1285d9c41df841a226d149eaf57, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=317, earliestPutTs=1681606434161 2023-04-16 00:53:56,839 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.Compactor(207): Compacting 4f0d2eafb2174fa495279f45b84c02d5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=353, earliestPutTs=1681606434280 2023-04-16 00:53:56,840 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.Compactor(207): Compacting 68fe0c007b144c26af83d9c2ef895eec, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=368, earliestPutTs=1681606434911 2023-04-16 00:53:56,842 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in c9039369968c422a52a2c7db83b430ba/B of c9039369968c422a52a2c7db83b430ba into 9a5fce57a03746c4910f4096856433c3(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 00:53:56,842 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for c9039369968c422a52a2c7db83b430ba: 2023-04-16 00:53:56,842 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba., storeName=c9039369968c422a52a2c7db83b430ba/B, priority=13, startTime=1681606436357; duration=0sec 2023-04-16 00:53:56,842 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 00:53:56,877 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] throttle.PressureAwareThroughputController(145): c9039369968c422a52a2c7db83b430ba#C#compaction#68 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-16 00:53:56,956 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:53:56,956 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40046 deadline: 1681606496955, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:57,035 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:53:57,036 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39990 deadline: 1681606497033, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:57,047 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:53:57,047 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39980 deadline: 1681606497047, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:57,048 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:53:57,049 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 183 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40044 deadline: 1681606497048, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:57,061 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:53:57,062 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40000 deadline: 1681606497054, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:57,261 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:53:57,261 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40046 deadline: 1681606497261, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:57,311 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=388 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/.tmp/C/5cda4afcb1f8485ab1ef5c121276c90c 2023-04-16 00:53:57,317 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/.tmp/A/5306221b1e9e44fb8abfe2f45f1bb09d as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/A/5306221b1e9e44fb8abfe2f45f1bb09d 2023-04-16 00:53:57,319 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/.tmp/C/5d8b41609f4e42a18e4d23e8c688880c as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/C/5d8b41609f4e42a18e4d23e8c688880c 2023-04-16 00:53:57,325 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/A/5306221b1e9e44fb8abfe2f45f1bb09d, entries=150, sequenceid=388, filesize=11.9 K 2023-04-16 00:53:57,326 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/.tmp/B/a5d40af55a374c5cb2763ce0ba9afb42 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/B/a5d40af55a374c5cb2763ce0ba9afb42 2023-04-16 00:53:57,328 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in c9039369968c422a52a2c7db83b430ba/C of c9039369968c422a52a2c7db83b430ba into 5d8b41609f4e42a18e4d23e8c688880c(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 00:53:57,328 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for c9039369968c422a52a2c7db83b430ba: 2023-04-16 00:53:57,328 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba., storeName=c9039369968c422a52a2c7db83b430ba/C, priority=13, startTime=1681606436357; duration=0sec 2023-04-16 00:53:57,328 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 00:53:57,332 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/B/a5d40af55a374c5cb2763ce0ba9afb42, entries=150, sequenceid=388, filesize=11.9 K 2023-04-16 00:53:57,333 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/.tmp/C/5cda4afcb1f8485ab1ef5c121276c90c as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/C/5cda4afcb1f8485ab1ef5c121276c90c 2023-04-16 00:53:57,340 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/C/5cda4afcb1f8485ab1ef5c121276c90c, entries=150, sequenceid=388, filesize=11.9 K 2023-04-16 00:53:57,341 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for c9039369968c422a52a2c7db83b430ba in 964ms, sequenceid=388, compaction requested=false 2023-04-16 00:53:57,341 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for c9039369968c422a52a2c7db83b430ba: 2023-04-16 00:53:57,342 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. 2023-04-16 00:53:57,342 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:53:57,342 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:53:57,342 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:57,342 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:53:57,342 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:53:57,343 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:57,344 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:57,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:57,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:53:57,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:53:57,344 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:57,344 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:53:57,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:53:57,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:53:57,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:57,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:57,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:53:57,346 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:53:57,346 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@26136c[Count = 0] remaining members to acquire global barrier 2023-04-16 00:53:57,346 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:53:57,346 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:57,347 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:57,347 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:57,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:57,347 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:53:57,347 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:53:57,347 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:53:57,347 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:57,347 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:53:57,349 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:57,349 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:53:57,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:57,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:53:57,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:53:57,349 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:53:57,349 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:53:57,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:53:57,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:53:57,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:57,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:57,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:53:57,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:57,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:57,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:53:57,354 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:53:57,354 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:53:57,354 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:53:57,354 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:53:57,354 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:53:57,354 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:53:57,357 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:57,357 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:57,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:57,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:53:57,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:53:57,357 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:57,357 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:53:57,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:57,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:53:57,358 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:53:57,358 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:57,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:53:57,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:57,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:57,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:53:57,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:57,359 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:57,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:57,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:53:57,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:57,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:57,364 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:57,364 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:57,364 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:53:57,364 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:57,364 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:53:57,364 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:53:57,364 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:57,364 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:57,364 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:57,364 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:57,364 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:53:57,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:53:57,364 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:53:57,365 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:53:57,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:53:57,365 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:53:57,365 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:53:57,364 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:53:57,466 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:53:57,466 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:53:57,468 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:53:57,469 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:53:57,469 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:53:57,469 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:53:57,469 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:53:57,469 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:53:57,469 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:53:57,470 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:57,470 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:57,505 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:57,505 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:53:57,505 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:53:57,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:53:57,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:57,507 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:57,507 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:53:57,507 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:57,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:53:57,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:57,507 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:53:57,508 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:53:57,508 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:53:57,508 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:53:57,508 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:53:57,508 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:53:57,508 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:53:57,508 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. 2023-04-16 00:53:57,508 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. started... 2023-04-16 00:53:57,509 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing c9039369968c422a52a2c7db83b430ba 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-16 00:53:57,509 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c9039369968c422a52a2c7db83b430ba, store=A 2023-04-16 00:53:57,509 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:53:57,509 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c9039369968c422a52a2c7db83b430ba, store=B 2023-04-16 00:53:57,509 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:53:57,509 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c9039369968c422a52a2c7db83b430ba, store=C 2023-04-16 00:53:57,509 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:53:57,538 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=403 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/.tmp/A/dcad1b35540143f6b8e7317070e485d0 2023-04-16 00:53:57,556 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=403 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/.tmp/B/bf0ec8695516431c893540f38a978dcc 2023-04-16 00:53:57,767 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(9158): Flush requested on c9039369968c422a52a2c7db83b430ba 2023-04-16 00:53:57,767 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. as already flushing 2023-04-16 00:53:57,816 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:53:57,816 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40046 deadline: 1681606497813, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:57,919 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:53:57,919 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 162 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40046 deadline: 1681606497918, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:57,971 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=403 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/.tmp/C/79bfe09a6a5f49d5a49988b3191a255f 2023-04-16 00:53:57,979 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/.tmp/A/dcad1b35540143f6b8e7317070e485d0 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/A/dcad1b35540143f6b8e7317070e485d0 2023-04-16 00:53:57,986 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/A/dcad1b35540143f6b8e7317070e485d0, entries=150, sequenceid=403, filesize=11.9 K 2023-04-16 00:53:57,987 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/.tmp/B/bf0ec8695516431c893540f38a978dcc as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/B/bf0ec8695516431c893540f38a978dcc 2023-04-16 00:53:57,999 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/B/bf0ec8695516431c893540f38a978dcc, entries=150, sequenceid=403, filesize=11.9 K 2023-04-16 00:53:58,000 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/.tmp/C/79bfe09a6a5f49d5a49988b3191a255f as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/C/79bfe09a6a5f49d5a49988b3191a255f 2023-04-16 00:53:58,005 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/C/79bfe09a6a5f49d5a49988b3191a255f, entries=150, sequenceid=403, filesize=11.9 K 2023-04-16 00:53:58,006 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for c9039369968c422a52a2c7db83b430ba in 498ms, sequenceid=403, compaction requested=true 2023-04-16 00:53:58,006 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for c9039369968c422a52a2c7db83b430ba: 2023-04-16 00:53:58,006 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. 2023-04-16 00:53:58,006 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:53:58,007 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:53:58,007 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:58,007 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:53:58,007 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:53:58,008 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:58,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:58,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:53:58,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:53:58,009 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:58,009 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:58,009 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:53:58,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:53:58,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:53:58,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:58,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:58,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:53:58,011 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:53:58,011 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@60aca501[Count = 0] remaining members to acquire global barrier 2023-04-16 00:53:58,011 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:53:58,011 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:58,012 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:58,013 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:58,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:58,013 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:53:58,013 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:53:58,013 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:53:58,013 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:58,013 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:53:58,015 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:58,015 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:53:58,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:58,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:53:58,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:53:58,015 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:53:58,015 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:53:58,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:53:58,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:53:58,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:58,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:58,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:53:58,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:58,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:58,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:53:58,018 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:53:58,018 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:53:58,018 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:53:58,018 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:53:58,018 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:53:58,018 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:53:58,019 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:58,020 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:58,020 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:58,020 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:53:58,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:58,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:58,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:53:58,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:53:58,020 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:53:58,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:53:58,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:53:58,021 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:58,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:58,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:58,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:53:58,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:58,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:58,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:53:58,023 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:58,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:58,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:58,025 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:58,025 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:53:58,025 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:58,025 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:53:58,026 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:53:58,025 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:58,025 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:53:58,025 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:53:58,025 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:53:58,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:53:58,026 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:58,026 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:53:58,026 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:58,026 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:53:58,026 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:58,026 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:58,026 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:53:58,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:53:58,124 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing c9039369968c422a52a2c7db83b430ba 3/3 column families, dataSize=120.76 KB heapSize=360.05 KB 2023-04-16 00:53:58,124 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c9039369968c422a52a2c7db83b430ba, store=A 2023-04-16 00:53:58,124 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 00:53:58,124 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c9039369968c422a52a2c7db83b430ba, store=B 2023-04-16 00:53:58,124 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 00:53:58,124 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c9039369968c422a52a2c7db83b430ba, store=C 2023-04-16 00:53:58,124 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 00:53:58,125 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(9158): Flush requested on c9039369968c422a52a2c7db83b430ba 2023-04-16 00:53:58,126 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:53:58,127 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:53:58,129 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:53:58,140 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:53:58,140 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:53:58,140 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:53:58,140 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:53:58,140 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:53:58,140 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:53:58,141 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:58,141 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:58,142 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:58,142 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:53:58,142 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:53:58,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:53:58,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:58,143 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:58,143 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:53:58,144 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:58,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:53:58,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:58,144 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:53:58,156 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:53:58,156 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:53:58,156 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:53:58,156 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:53:58,156 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:53:58,160 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:53:58,160 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:53:58,162 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. 2023-04-16 00:53:58,162 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. started... 2023-04-16 00:53:58,163 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. as already flushing 2023-04-16 00:53:58,163 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. 2023-04-16 00:53:58,163 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:53:58,163 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:53:58,163 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:58,163 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:53:58,163 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:53:58,165 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:53:58,166 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40046 deadline: 1681606498164, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:58,166 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:58,166 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:58,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:58,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:53:58,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:53:58,167 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:58,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:53:58,167 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:53:58,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:53:58,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:58,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:58,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:53:58,168 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:53:58,168 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@48d260a9[Count = 0] remaining members to acquire global barrier 2023-04-16 00:53:58,168 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:53:58,168 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:58,170 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:58,170 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:58,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:58,170 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:53:58,170 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:53:58,170 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:53:58,171 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:58,171 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:53:58,172 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:53:58,172 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:53:58,172 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:53:58,172 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:58,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:58,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:53:58,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:53:58,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:53:58,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:53:58,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:58,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:58,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:53:58,175 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=424 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/.tmp/A/ba06e0f624084caf83908a6a174a963e 2023-04-16 00:53:58,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:58,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:58,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:53:58,178 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:53:58,178 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:53:58,179 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:53:58,179 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:53:58,179 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:53:58,179 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:53:58,182 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:58,182 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:58,182 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:53:58,182 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:58,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:58,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:58,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:53:58,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:53:58,183 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:53:58,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:53:58,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:58,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:53:58,185 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:58,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:58,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:53:58,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:58,187 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:58,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:58,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:53:58,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:58,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:58,201 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:53:58,201 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:58,201 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:53:58,201 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:53:58,201 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:58,201 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:58,201 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:58,201 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:53:58,201 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:53:58,201 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:53:58,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:53:58,201 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:53:58,201 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:53:58,201 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:58,201 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:53:58,201 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:58,201 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:58,202 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:53:58,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:53:58,268 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:53:58,268 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 175 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40046 deadline: 1681606498268, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:58,301 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:53:58,302 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:53:58,304 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:53:58,305 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:53:58,305 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:53:58,305 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:53:58,305 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:53:58,305 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:53:58,305 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:53:58,306 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:58,306 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:58,307 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:53:58,307 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:58,307 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:53:58,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:53:58,307 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:58,307 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:53:58,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:58,308 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:58,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:53:58,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:58,309 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:53:58,309 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:53:58,309 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:53:58,310 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:53:58,310 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:53:58,310 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:53:58,310 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:53:58,310 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:53:58,310 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. 2023-04-16 00:53:58,310 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. started... 2023-04-16 00:53:58,310 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. as already flushing 2023-04-16 00:53:58,310 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. 2023-04-16 00:53:58,310 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:53:58,310 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:53:58,310 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:58,310 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:53:58,310 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:53:58,312 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:58,312 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:58,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:58,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:53:58,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:53:58,312 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:58,312 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:53:58,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:53:58,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:53:58,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:58,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:58,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:53:58,314 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:53:58,314 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@626e150b[Count = 0] remaining members to acquire global barrier 2023-04-16 00:53:58,314 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:53:58,314 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:58,315 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:58,315 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:58,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:58,315 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:53:58,315 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:53:58,315 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:53:58,315 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:58,315 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:53:58,317 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:58,317 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:53:58,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:58,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:53:58,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:53:58,317 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:53:58,317 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:53:58,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:53:58,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:53:58,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:58,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:58,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:53:58,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:58,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:58,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:53:58,320 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:53:58,320 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:53:58,320 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:53:58,320 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:53:58,320 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:53:58,320 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:53:58,322 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:58,322 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:53:58,322 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:58,322 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:58,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:58,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:58,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:53:58,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:53:58,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:53:58,323 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:53:58,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:53:58,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:58,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:58,334 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:58,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:53:58,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:58,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:58,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:53:58,335 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:58,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:58,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:58,338 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:58,338 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:58,338 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:58,338 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:58,338 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:53:58,338 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:53:58,338 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:53:58,338 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:53:58,338 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:58,338 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:53:58,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:53:58,338 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:53:58,338 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:53:58,338 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:53:58,338 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:58,338 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:58,339 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:53:58,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:53:58,438 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:53:58,439 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:53:58,441 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:53:58,441 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:53:58,442 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:53:58,442 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:53:58,442 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:53:58,442 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:53:58,442 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:53:58,442 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:58,442 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:58,444 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:53:58,444 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:58,444 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:53:58,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:53:58,444 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:58,444 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:53:58,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:58,445 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:58,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:53:58,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:58,445 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:53:58,445 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:53:58,445 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:53:58,445 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:53:58,446 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:53:58,446 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:53:58,446 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:53:58,446 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:53:58,446 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. 2023-04-16 00:53:58,446 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. started... 2023-04-16 00:53:58,446 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. as already flushing 2023-04-16 00:53:58,446 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. 2023-04-16 00:53:58,446 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:53:58,446 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:53:58,446 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:58,446 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:53:58,446 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:53:58,449 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:58,449 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:58,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:58,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:53:58,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:53:58,449 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:58,449 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:53:58,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:53:58,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:53:58,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:58,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:58,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:53:58,451 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:53:58,451 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5263d759[Count = 0] remaining members to acquire global barrier 2023-04-16 00:53:58,451 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:53:58,451 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:58,452 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:58,453 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:58,453 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:53:58,457 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:58,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:58,457 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:53:58,457 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:53:58,457 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:53:58,459 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:58,459 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:53:58,459 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:53:58,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:58,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:53:58,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:53:58,459 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:53:58,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:53:58,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:53:58,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:58,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:58,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:53:58,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:58,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:58,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:53:58,463 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:53:58,463 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:53:58,463 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:53:58,463 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:53:58,463 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:53:58,463 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:53:58,464 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:58,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:58,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:53:58,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:53:58,465 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:58,465 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:53:58,465 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:58,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:58,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:53:58,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:58,465 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:58,465 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:53:58,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:53:58,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:53:58,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:58,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:58,471 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:53:58,471 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 177 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40046 deadline: 1681606498470, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:58,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:58,472 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:58,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:53:58,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:58,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:58,479 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:53:58,479 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:58,479 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:53:58,479 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:53:58,479 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:53:58,479 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:53:58,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:53:58,479 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:53:58,479 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:58,479 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:58,479 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:53:58,479 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:53:58,479 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:58,480 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:58,480 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:58,480 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:58,480 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:53:58,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:53:58,579 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:53:58,580 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:53:58,582 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:53:58,583 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:53:58,583 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:53:58,583 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:53:58,583 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:53:58,583 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:53:58,583 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:53:58,584 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:58,584 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:58,585 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:53:58,585 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:58,585 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:53:58,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:53:58,586 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:58,586 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:53:58,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:58,586 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:58,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:53:58,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:58,587 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:53:58,587 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:53:58,587 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:53:58,587 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:53:58,587 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:53:58,587 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:53:58,588 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:53:58,588 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:53:58,588 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. 2023-04-16 00:53:58,588 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. started... 2023-04-16 00:53:58,588 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. as already flushing 2023-04-16 00:53:58,588 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. 2023-04-16 00:53:58,588 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:53:58,588 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:53:58,588 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:58,588 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:53:58,588 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:53:58,590 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:58,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:58,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:53:58,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:53:58,590 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:58,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:53:58,591 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:58,591 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:53:58,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:53:58,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:58,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:58,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:53:58,592 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:53:58,592 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@57e2e4b9[Count = 0] remaining members to acquire global barrier 2023-04-16 00:53:58,592 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:53:58,592 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:58,593 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:58,593 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:58,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:58,593 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:53:58,593 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:53:58,593 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:53:58,593 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:58,593 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:53:58,595 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:58,595 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:53:58,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:58,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:53:58,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:53:58,595 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:53:58,596 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:53:58,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:53:58,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:53:58,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:58,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:58,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:53:58,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:58,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:58,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:53:58,599 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:53:58,599 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:53:58,599 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:53:58,599 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:53:58,599 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:53:58,599 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:53:58,601 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:58,601 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:58,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:58,601 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:58,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:53:58,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:53:58,601 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:53:58,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:58,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:53:58,601 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:58,601 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:53:58,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:53:58,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:58,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:58,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:53:58,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:58,603 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:58,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:58,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:53:58,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:58,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:58,607 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:53:58,607 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:53:58,607 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:53:58,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:53:58,607 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:58,608 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:58,608 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:53:58,608 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:53:58,608 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:53:58,608 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:58,608 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:53:58,608 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:58,608 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:58,608 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:58,608 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:58,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:53:58,608 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:53:58,608 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:53:58,644 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=424 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/.tmp/B/4fc9287e9d5d4d1795c97bbff98cd5de 2023-04-16 00:53:58,708 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:53:58,709 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:53:58,711 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:53:58,718 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:53:58,718 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:53:58,718 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:53:58,719 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:53:58,719 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:53:58,719 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:53:58,719 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:58,719 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:58,720 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:58,720 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:53:58,721 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:53:58,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:53:58,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:58,722 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=424 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/.tmp/C/c2040e6802c44108be891184e85171ed 2023-04-16 00:53:58,722 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:58,722 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:53:58,722 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:58,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:53:58,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:58,723 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:53:58,723 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:53:58,723 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:53:58,723 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:53:58,723 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:53:58,723 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:53:58,726 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:53:58,726 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:53:58,726 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. 2023-04-16 00:53:58,726 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. started... 2023-04-16 00:53:58,726 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. as already flushing 2023-04-16 00:53:58,726 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. 2023-04-16 00:53:58,726 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:53:58,726 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:53:58,726 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:58,726 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:53:58,726 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:53:58,728 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:58,728 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:58,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:58,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:53:58,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:53:58,728 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:58,728 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:53:58,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:53:58,729 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/.tmp/A/ba06e0f624084caf83908a6a174a963e as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/A/ba06e0f624084caf83908a6a174a963e 2023-04-16 00:53:58,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:53:58,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:58,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:58,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:53:58,731 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:53:58,731 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@c6e01c4[Count = 0] remaining members to acquire global barrier 2023-04-16 00:53:58,731 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:53:58,731 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:58,733 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:58,733 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:58,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:58,733 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:53:58,733 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:53:58,733 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:53:58,733 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:58,733 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:53:58,734 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:58,734 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:53:58,734 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:53:58,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:58,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:53:58,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:53:58,735 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:53:58,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:53:58,735 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/A/ba06e0f624084caf83908a6a174a963e, entries=150, sequenceid=424, filesize=11.9 K 2023-04-16 00:53:58,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:53:58,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:58,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:58,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:53:58,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:58,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:58,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:53:58,738 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:53:58,738 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:53:58,738 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:53:58,738 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:53:58,738 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:53:58,738 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:53:58,739 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/.tmp/B/4fc9287e9d5d4d1795c97bbff98cd5de as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/B/4fc9287e9d5d4d1795c97bbff98cd5de 2023-04-16 00:53:58,745 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:58,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:58,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:53:58,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:53:58,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:53:58,745 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:58,745 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:58,746 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:53:58,746 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:58,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:58,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:58,746 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:53:58,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:53:58,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:58,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:53:58,749 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:58,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:58,753 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/B/4fc9287e9d5d4d1795c97bbff98cd5de, entries=150, sequenceid=424, filesize=11.9 K 2023-04-16 00:53:58,754 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/.tmp/C/c2040e6802c44108be891184e85171ed as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/C/c2040e6802c44108be891184e85171ed 2023-04-16 00:53:58,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:58,761 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/C/c2040e6802c44108be891184e85171ed, entries=150, sequenceid=424, filesize=11.9 K 2023-04-16 00:53:58,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:53:58,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:58,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:58,762 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~120.76 KB/123660, heapSize ~360 KB/368640, currentSize=53.67 KB/54960 for c9039369968c422a52a2c7db83b430ba in 638ms, sequenceid=424, compaction requested=true 2023-04-16 00:53:58,762 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for c9039369968c422a52a2c7db83b430ba: 2023-04-16 00:53:58,762 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 00:53:58,762 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-16 00:53:58,762 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-16 00:53:58,762 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 00:53:58,762 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 00:53:58,763 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:53:58,763 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:53:58,763 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:58,763 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:58,763 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:58,763 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:53:58,763 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:53:58,763 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:58,763 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:53:58,763 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:53:58,763 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:58,763 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:58,763 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:58,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:53:58,764 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:53:58,764 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:53:58,764 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:53:58,764 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49460 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-16 00:53:58,764 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:53:58,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:53:58,764 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HStore(1912): c9039369968c422a52a2c7db83b430ba/A is initiating minor compaction (all files) 2023-04-16 00:53:58,764 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of c9039369968c422a52a2c7db83b430ba/A in TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. 2023-04-16 00:53:58,764 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/A/956450698f4741c890069edfacfe3aed, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/A/5306221b1e9e44fb8abfe2f45f1bb09d, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/A/dcad1b35540143f6b8e7317070e485d0, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/A/ba06e0f624084caf83908a6a174a963e] into tmpdir=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/.tmp, totalSize=48.3 K 2023-04-16 00:53:58,765 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49460 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-16 00:53:58,765 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HStore(1912): c9039369968c422a52a2c7db83b430ba/B is initiating minor compaction (all files) 2023-04-16 00:53:58,765 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HRegion(2259): Starting compaction of c9039369968c422a52a2c7db83b430ba/B in TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. 2023-04-16 00:53:58,765 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/B/9a5fce57a03746c4910f4096856433c3, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/B/a5d40af55a374c5cb2763ce0ba9afb42, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/B/bf0ec8695516431c893540f38a978dcc, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/B/4fc9287e9d5d4d1795c97bbff98cd5de] into tmpdir=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/.tmp, totalSize=48.3 K 2023-04-16 00:53:58,766 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting 9a5fce57a03746c4910f4096856433c3, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=368, earliestPutTs=1681606434911 2023-04-16 00:53:58,767 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.Compactor(207): Compacting 956450698f4741c890069edfacfe3aed, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=368, earliestPutTs=1681606434911 2023-04-16 00:53:58,767 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting a5d40af55a374c5cb2763ce0ba9afb42, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=388, earliestPutTs=1681606435478 2023-04-16 00:53:58,767 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.Compactor(207): Compacting 5306221b1e9e44fb8abfe2f45f1bb09d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=388, earliestPutTs=1681606435478 2023-04-16 00:53:58,767 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting bf0ec8695516431c893540f38a978dcc, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=403, earliestPutTs=1681606436640 2023-04-16 00:53:58,768 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.Compactor(207): Compacting dcad1b35540143f6b8e7317070e485d0, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=403, earliestPutTs=1681606436640 2023-04-16 00:53:58,768 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting 4fc9287e9d5d4d1795c97bbff98cd5de, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=424, earliestPutTs=1681606437810 2023-04-16 00:53:58,769 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.Compactor(207): Compacting ba06e0f624084caf83908a6a174a963e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=424, earliestPutTs=1681606437810 2023-04-16 00:53:58,790 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(9158): Flush requested on c9039369968c422a52a2c7db83b430ba 2023-04-16 00:53:58,790 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing c9039369968c422a52a2c7db83b430ba 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-16 00:53:58,790 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c9039369968c422a52a2c7db83b430ba, store=A 2023-04-16 00:53:58,790 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:53:58,791 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c9039369968c422a52a2c7db83b430ba, store=B 2023-04-16 00:53:58,791 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:53:58,791 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c9039369968c422a52a2c7db83b430ba, store=C 2023-04-16 00:53:58,791 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:53:58,797 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] throttle.PressureAwareThroughputController(145): c9039369968c422a52a2c7db83b430ba#A#compaction#75 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-16 00:53:58,814 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] throttle.PressureAwareThroughputController(145): c9039369968c422a52a2c7db83b430ba#B#compaction#77 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-16 00:53:58,864 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=436 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/.tmp/A/69a44398ac3e4a729eb1570d447713b6 2023-04-16 00:53:58,864 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:53:58,865 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:53:58,867 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:53:58,871 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/.tmp/A/9755bbfe9d7c4bac80b06b65f45845ad as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/A/9755bbfe9d7c4bac80b06b65f45845ad 2023-04-16 00:53:58,879 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:53:58,879 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:53:58,879 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:53:58,879 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:53:58,879 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:53:58,879 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:53:58,880 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:58,880 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:58,881 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:53:58,881 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:58,881 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:53:58,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:53:58,882 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:58,882 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:53:58,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:58,882 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:58,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:53:58,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:58,883 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:53:58,883 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in c9039369968c422a52a2c7db83b430ba/A of c9039369968c422a52a2c7db83b430ba into 9755bbfe9d7c4bac80b06b65f45845ad(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 00:53:58,883 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for c9039369968c422a52a2c7db83b430ba: 2023-04-16 00:53:58,883 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba., storeName=c9039369968c422a52a2c7db83b430ba/A, priority=12, startTime=1681606438762; duration=0sec 2023-04-16 00:53:58,883 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 00:53:58,883 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-16 00:53:58,886 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49460 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-16 00:53:58,886 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HStore(1912): c9039369968c422a52a2c7db83b430ba/C is initiating minor compaction (all files) 2023-04-16 00:53:58,886 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of c9039369968c422a52a2c7db83b430ba/C in TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. 2023-04-16 00:53:58,886 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/C/5d8b41609f4e42a18e4d23e8c688880c, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/C/5cda4afcb1f8485ab1ef5c121276c90c, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/C/79bfe09a6a5f49d5a49988b3191a255f, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/C/c2040e6802c44108be891184e85171ed] into tmpdir=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/.tmp, totalSize=48.3 K 2023-04-16 00:53:58,887 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.Compactor(207): Compacting 5d8b41609f4e42a18e4d23e8c688880c, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=368, earliestPutTs=1681606434911 2023-04-16 00:53:58,887 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.Compactor(207): Compacting 5cda4afcb1f8485ab1ef5c121276c90c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=388, earliestPutTs=1681606435478 2023-04-16 00:53:58,888 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.Compactor(207): Compacting 79bfe09a6a5f49d5a49988b3191a255f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=403, earliestPutTs=1681606436640 2023-04-16 00:53:58,888 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.Compactor(207): Compacting c2040e6802c44108be891184e85171ed, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=424, earliestPutTs=1681606437810 2023-04-16 00:53:58,914 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/.tmp/B/bb7b26c820ee462182b077a56191b08d as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/B/bb7b26c820ee462182b077a56191b08d 2023-04-16 00:53:58,916 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:53:58,916 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:53:58,916 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:53:58,916 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:53:58,916 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:53:58,916 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:53:58,916 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:53:58,917 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. 2023-04-16 00:53:58,917 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. started... 2023-04-16 00:53:58,917 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. as already flushing 2023-04-16 00:53:58,917 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. 2023-04-16 00:53:58,917 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:53:58,917 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:53:58,917 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:58,917 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:53:58,917 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:53:58,920 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:58,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:58,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:53:58,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:53:58,921 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:58,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:53:58,921 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:58,921 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:53:58,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:53:58,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:58,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:58,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:53:58,924 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:53:58,925 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1eba9b0f[Count = 0] remaining members to acquire global barrier 2023-04-16 00:53:58,925 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in c9039369968c422a52a2c7db83b430ba/B of c9039369968c422a52a2c7db83b430ba into bb7b26c820ee462182b077a56191b08d(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 00:53:58,925 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:53:58,925 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for c9039369968c422a52a2c7db83b430ba: 2023-04-16 00:53:58,925 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:58,925 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba., storeName=c9039369968c422a52a2c7db83b430ba/B, priority=12, startTime=1681606438762; duration=0sec 2023-04-16 00:53:58,925 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 00:53:58,926 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:58,926 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:58,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:58,926 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:53:58,926 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:53:58,926 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:53:58,928 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:58,928 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:53:58,929 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:58,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:58,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:53:58,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:53:58,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:53:58,930 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] throttle.PressureAwareThroughputController(145): c9039369968c422a52a2c7db83b430ba#C#compaction#79 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-16 00:53:58,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:53:58,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:58,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:58,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:53:58,931 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:53:58,931 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:53:58,931 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:53:58,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:58,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:58,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:53:58,933 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:53:58,934 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:53:58,934 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:53:58,934 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:53:58,934 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:53:58,934 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:53:58,936 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:58,936 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:58,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:58,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:53:58,936 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:58,936 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:53:58,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:58,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:53:58,936 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:53:58,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:53:58,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:53:58,936 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:58,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:58,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:58,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:53:58,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:58,938 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:58,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:58,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:53:58,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:58,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:58,944 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:53:58,944 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 197 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40046 deadline: 1681606498932, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:58,946 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:53:58,946 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:53:58,946 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:58,946 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:58,946 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:53:58,946 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:53:58,946 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:53:58,946 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:53:58,946 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:53:58,946 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:58,946 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:58,946 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:58,946 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:58,946 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:58,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:53:58,946 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:53:58,947 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:53:58,947 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=436 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/.tmp/B/ff34a3c646df407f8add805a5f987768 2023-04-16 00:53:58,947 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:53:58,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:53:58,984 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/.tmp/C/afb3b38579dc4f0fb532aa79bf8ca79c as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/C/afb3b38579dc4f0fb532aa79bf8ca79c 2023-04-16 00:53:58,991 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in c9039369968c422a52a2c7db83b430ba/C of c9039369968c422a52a2c7db83b430ba into afb3b38579dc4f0fb532aa79bf8ca79c(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 00:53:58,991 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for c9039369968c422a52a2c7db83b430ba: 2023-04-16 00:53:58,991 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba., storeName=c9039369968c422a52a2c7db83b430ba/C, priority=12, startTime=1681606438762; duration=0sec 2023-04-16 00:53:58,991 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 00:53:59,008 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=436 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/.tmp/C/64e168accb044b7aa27fa8331d3872fb 2023-04-16 00:53:59,016 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/.tmp/A/69a44398ac3e4a729eb1570d447713b6 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/A/69a44398ac3e4a729eb1570d447713b6 2023-04-16 00:53:59,023 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/A/69a44398ac3e4a729eb1570d447713b6, entries=200, sequenceid=436, filesize=14.3 K 2023-04-16 00:53:59,024 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/.tmp/B/ff34a3c646df407f8add805a5f987768 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/B/ff34a3c646df407f8add805a5f987768 2023-04-16 00:53:59,032 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/B/ff34a3c646df407f8add805a5f987768, entries=150, sequenceid=436, filesize=11.9 K 2023-04-16 00:53:59,033 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/.tmp/C/64e168accb044b7aa27fa8331d3872fb as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/C/64e168accb044b7aa27fa8331d3872fb 2023-04-16 00:53:59,047 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:53:59,048 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:53:59,048 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39990 deadline: 1681606499043, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:59,048 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:53:59,048 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 199 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40046 deadline: 1681606499045, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:59,049 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:53:59,056 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/C/64e168accb044b7aa27fa8331d3872fb, entries=150, sequenceid=436, filesize=11.9 K 2023-04-16 00:53:59,057 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:53:59,058 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for c9039369968c422a52a2c7db83b430ba in 268ms, sequenceid=436, compaction requested=false 2023-04-16 00:53:59,059 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for c9039369968c422a52a2c7db83b430ba: 2023-04-16 00:53:59,064 DEBUG [Thread-149] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4140 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba., hostname=jenkins-hbase4.apache.org,34975,1681606419217, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor32.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-16 00:53:59,070 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(9158): Flush requested on c9039369968c422a52a2c7db83b430ba 2023-04-16 00:53:59,072 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681606438904, max timestamp=1681606439069Num uniques 859; 2023-04-16 00:53:59,072 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681606438904, max timestamp=1681606439069Num uniques 859; 2023-04-16 00:53:59,072 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-16 00:53:59,073 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681606438904, max timestamp=1681606439069Num uniques 859; 2023-04-16 00:53:59,073 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-16 00:53:59,072 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-16 00:53:59,084 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing c9039369968c422a52a2c7db83b430ba 3/3 column families, dataSize=60.38 KB heapSize=165.84 KB 2023-04-16 00:53:59,085 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c9039369968c422a52a2c7db83b430ba, store=A 2023-04-16 00:53:59,086 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 00:53:59,086 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c9039369968c422a52a2c7db83b430ba, store=B 2023-04-16 00:53:59,086 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 00:53:59,086 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c9039369968c422a52a2c7db83b430ba, store=C 2023-04-16 00:53:59,086 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 00:53:59,088 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:53:59,088 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:53:59,088 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:53:59,088 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:53:59,088 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:53:59,088 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:53:59,089 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:59,089 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:59,091 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:59,091 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:53:59,091 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:53:59,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:53:59,091 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:59,091 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:53:59,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:59,092 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:59,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:53:59,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:59,092 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:53:59,092 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:53:59,092 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:53:59,093 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:53:59,093 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:53:59,093 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:53:59,093 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:53:59,093 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:53:59,093 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. 2023-04-16 00:53:59,093 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. started... 2023-04-16 00:53:59,093 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. as already flushing 2023-04-16 00:53:59,093 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. 2023-04-16 00:53:59,093 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:53:59,093 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:53:59,094 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:59,094 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:53:59,094 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:53:59,096 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:59,096 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:59,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:59,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:53:59,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:53:59,097 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:59,097 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:53:59,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:53:59,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:53:59,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:59,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:59,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:53:59,100 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:53:59,100 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@52af4e9e[Count = 0] remaining members to acquire global barrier 2023-04-16 00:53:59,100 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:53:59,100 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:59,106 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:59,106 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:59,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:59,106 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:53:59,106 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:53:59,106 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:59,106 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:53:59,106 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:53:59,108 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:53:59,108 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:59,108 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:53:59,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:59,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:53:59,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:53:59,108 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:53:59,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:53:59,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:53:59,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:59,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:59,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:53:59,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:59,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:59,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:53:59,112 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:53:59,112 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:53:59,112 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:53:59,112 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:53:59,112 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:53:59,112 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:53:59,113 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:59,113 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:59,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:59,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:53:59,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:53:59,113 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:53:59,113 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:59,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:59,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:53:59,114 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:53:59,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:53:59,114 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:59,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:59,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:59,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:53:59,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:59,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:59,116 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:59,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:53:59,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:59,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:59,121 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:59,121 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:59,121 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:59,121 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:53:59,121 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:53:59,121 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:53:59,121 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:59,121 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:53:59,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:53:59,121 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:53:59,122 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:53:59,122 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:59,122 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:59,122 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:59,122 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:53:59,122 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:53:59,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:53:59,122 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:53:59,125 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=466 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/.tmp/A/d00e67b4373d45969586af9c48ccd449 2023-04-16 00:53:59,129 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:53:59,130 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 193 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40044 deadline: 1681606499125, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:59,131 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:53:59,131 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39980 deadline: 1681606499127, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:59,132 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:53:59,132 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 177 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40000 deadline: 1681606499128, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:59,159 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=466 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/.tmp/B/4fa4401a18944b888a1d4402241493b3 2023-04-16 00:53:59,180 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=466 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/.tmp/C/a4d87411c83443929fdbe9b819ec40a4 2023-04-16 00:53:59,187 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/.tmp/A/d00e67b4373d45969586af9c48ccd449 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/A/d00e67b4373d45969586af9c48ccd449 2023-04-16 00:53:59,193 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/A/d00e67b4373d45969586af9c48ccd449, entries=150, sequenceid=466, filesize=11.9 K 2023-04-16 00:53:59,195 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/.tmp/B/4fa4401a18944b888a1d4402241493b3 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/B/4fa4401a18944b888a1d4402241493b3 2023-04-16 00:53:59,208 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/B/4fa4401a18944b888a1d4402241493b3, entries=150, sequenceid=466, filesize=11.9 K 2023-04-16 00:53:59,210 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/.tmp/C/a4d87411c83443929fdbe9b819ec40a4 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/C/a4d87411c83443929fdbe9b819ec40a4 2023-04-16 00:53:59,215 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/C/a4d87411c83443929fdbe9b819ec40a4, entries=150, sequenceid=466, filesize=11.9 K 2023-04-16 00:53:59,217 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~185.72 KB/190176, currentSize=114.05 KB/116790 for c9039369968c422a52a2c7db83b430ba in 133ms, sequenceid=466, compaction requested=true 2023-04-16 00:53:59,217 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for c9039369968c422a52a2c7db83b430ba: 2023-04-16 00:53:59,217 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-16 00:53:59,218 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 00:53:59,218 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 00:53:59,218 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 00:53:59,218 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-16 00:53:59,218 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 39850 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-16 00:53:59,219 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HStore(1912): c9039369968c422a52a2c7db83b430ba/A is initiating minor compaction (all files) 2023-04-16 00:53:59,219 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of c9039369968c422a52a2c7db83b430ba/A in TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. 2023-04-16 00:53:59,219 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/A/9755bbfe9d7c4bac80b06b65f45845ad, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/A/69a44398ac3e4a729eb1570d447713b6, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/A/d00e67b4373d45969586af9c48ccd449] into tmpdir=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/.tmp, totalSize=38.9 K 2023-04-16 00:53:59,219 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37410 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-16 00:53:59,219 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HStore(1912): c9039369968c422a52a2c7db83b430ba/B is initiating minor compaction (all files) 2023-04-16 00:53:59,220 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HRegion(2259): Starting compaction of c9039369968c422a52a2c7db83b430ba/B in TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. 2023-04-16 00:53:59,220 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/B/bb7b26c820ee462182b077a56191b08d, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/B/ff34a3c646df407f8add805a5f987768, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/B/4fa4401a18944b888a1d4402241493b3] into tmpdir=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/.tmp, totalSize=36.5 K 2023-04-16 00:53:59,220 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.Compactor(207): Compacting 9755bbfe9d7c4bac80b06b65f45845ad, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=424, earliestPutTs=1681606437810 2023-04-16 00:53:59,220 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting bb7b26c820ee462182b077a56191b08d, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=424, earliestPutTs=1681606437810 2023-04-16 00:53:59,220 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.Compactor(207): Compacting 69a44398ac3e4a729eb1570d447713b6, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=436, earliestPutTs=1681606438128 2023-04-16 00:53:59,221 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting ff34a3c646df407f8add805a5f987768, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=436, earliestPutTs=1681606438128 2023-04-16 00:53:59,221 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting 4fa4401a18944b888a1d4402241493b3, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=466, earliestPutTs=1681606439073 2023-04-16 00:53:59,222 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:53:59,223 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:53:59,225 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:53:59,227 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.Compactor(207): Compacting d00e67b4373d45969586af9c48ccd449, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=466, earliestPutTs=1681606439073 2023-04-16 00:53:59,228 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:53:59,228 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:53:59,228 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:53:59,228 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:53:59,228 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:53:59,228 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:53:59,229 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:59,229 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:59,233 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:53:59,233 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:59,233 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:53:59,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:53:59,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:59,234 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:59,234 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:59,234 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:53:59,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:53:59,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:59,235 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:53:59,236 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(9158): Flush requested on c9039369968c422a52a2c7db83b430ba 2023-04-16 00:53:59,236 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681606439099, max timestamp=1681606439233Num uniques 859; 2023-04-16 00:53:59,236 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-16 00:53:59,236 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing c9039369968c422a52a2c7db83b430ba 3/3 column families, dataSize=89.45 KB heapSize=262.11 KB 2023-04-16 00:53:59,236 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681606439099, max timestamp=1681606439233Num uniques 859; 2023-04-16 00:53:59,237 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-16 00:53:59,237 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681606439099, max timestamp=1681606439233Num uniques 859; 2023-04-16 00:53:59,237 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-16 00:53:59,237 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c9039369968c422a52a2c7db83b430ba, store=A 2023-04-16 00:53:59,238 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 00:53:59,238 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c9039369968c422a52a2c7db83b430ba, store=B 2023-04-16 00:53:59,238 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 00:53:59,238 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c9039369968c422a52a2c7db83b430ba, store=C 2023-04-16 00:53:59,238 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 00:53:59,242 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:53:59,242 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:53:59,242 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:53:59,242 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:53:59,242 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:53:59,243 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:53:59,243 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:53:59,243 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] throttle.PressureAwareThroughputController(145): c9039369968c422a52a2c7db83b430ba#B#compaction#84 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-16 00:53:59,243 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. 2023-04-16 00:53:59,243 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. started... 2023-04-16 00:53:59,243 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. as already flushing 2023-04-16 00:53:59,243 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. 2023-04-16 00:53:59,243 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:53:59,243 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:53:59,243 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:59,243 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:53:59,243 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:53:59,245 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:59,245 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:59,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:59,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:53:59,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:53:59,245 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:59,245 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:53:59,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:53:59,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:53:59,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:59,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:59,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:53:59,247 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:53:59,247 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@12a8af5e[Count = 0] remaining members to acquire global barrier 2023-04-16 00:53:59,247 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:53:59,247 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:59,248 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:59,248 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:59,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:59,248 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:53:59,248 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:53:59,248 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:53:59,250 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] throttle.PressureAwareThroughputController(145): c9039369968c422a52a2c7db83b430ba#A#compaction#85 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-16 00:53:59,252 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:53:59,252 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:53:59,252 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:53:59,255 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:59,256 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:53:59,256 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:53:59,256 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:53:59,256 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:53:59,256 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:53:59,256 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:53:59,256 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:53:59,257 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:59,258 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:59,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:59,258 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:59,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:59,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:53:59,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:53:59,258 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:53:59,258 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:53:59,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:53:59,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:59,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:53:59,261 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:59,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:59,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:53:59,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:59,262 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:59,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:59,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:53:59,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:59,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:59,265 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:59,265 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:53:59,265 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:59,265 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:53:59,265 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:53:59,265 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:53:59,265 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:53:59,265 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:59,266 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:59,265 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:53:59,266 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:53:59,266 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:53:59,265 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:53:59,266 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:59,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:53:59,266 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:59,266 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:59,267 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:53:59,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:53:59,277 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1681606439260, max timestamp=1681606439273Num uniques 859; 2023-04-16 00:53:59,277 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1681606439260, max timestamp=1681606439273Num uniques 859; 2023-04-16 00:53:59,277 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-16 00:53:59,277 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1681606439260, max timestamp=1681606439273Num uniques 859; 2023-04-16 00:53:59,277 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-16 00:53:59,277 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-16 00:53:59,299 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/.tmp/B/54936fa0a5084f72b8636cdc748cb8af as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/B/54936fa0a5084f72b8636cdc748cb8af 2023-04-16 00:53:59,309 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in c9039369968c422a52a2c7db83b430ba/B of c9039369968c422a52a2c7db83b430ba into 54936fa0a5084f72b8636cdc748cb8af(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 00:53:59,309 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for c9039369968c422a52a2c7db83b430ba: 2023-04-16 00:53:59,309 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba., storeName=c9039369968c422a52a2c7db83b430ba/B, priority=13, startTime=1681606439218; duration=0sec 2023-04-16 00:53:59,309 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 00:53:59,309 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-16 00:53:59,311 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37410 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-16 00:53:59,311 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HStore(1912): c9039369968c422a52a2c7db83b430ba/C is initiating minor compaction (all files) 2023-04-16 00:53:59,312 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:53:59,312 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HRegion(2259): Starting compaction of c9039369968c422a52a2c7db83b430ba/C in TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. 2023-04-16 00:53:59,312 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/C/afb3b38579dc4f0fb532aa79bf8ca79c, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/C/64e168accb044b7aa27fa8331d3872fb, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/C/a4d87411c83443929fdbe9b819ec40a4] into tmpdir=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/.tmp, totalSize=36.5 K 2023-04-16 00:53:59,312 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 209 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40046 deadline: 1681606499305, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:59,313 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:53:59,313 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting afb3b38579dc4f0fb532aa79bf8ca79c, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=424, earliestPutTs=1681606437810 2023-04-16 00:53:59,313 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 188 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40000 deadline: 1681606499305, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:59,314 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting 64e168accb044b7aa27fa8331d3872fb, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=436, earliestPutTs=1681606438128 2023-04-16 00:53:59,317 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:53:59,317 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 205 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40044 deadline: 1681606499310, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:59,317 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:53:59,317 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39980 deadline: 1681606499312, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:59,321 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting a4d87411c83443929fdbe9b819ec40a4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=466, earliestPutTs=1681606439073 2023-04-16 00:53:59,333 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] throttle.PressureAwareThroughputController(145): c9039369968c422a52a2c7db83b430ba#C#compaction#87 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-16 00:53:59,360 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/.tmp/C/c0940893fadc4e2b8a120177cc1e09cc as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/C/c0940893fadc4e2b8a120177cc1e09cc 2023-04-16 00:53:59,366 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:53:59,367 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:53:59,369 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in c9039369968c422a52a2c7db83b430ba/C of c9039369968c422a52a2c7db83b430ba into c0940893fadc4e2b8a120177cc1e09cc(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 00:53:59,369 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for c9039369968c422a52a2c7db83b430ba: 2023-04-16 00:53:59,369 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba., storeName=c9039369968c422a52a2c7db83b430ba/C, priority=13, startTime=1681606439218; duration=0sec 2023-04-16 00:53:59,369 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 00:53:59,373 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:53:59,376 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:53:59,376 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:53:59,376 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:53:59,376 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:53:59,376 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:53:59,376 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:53:59,377 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:59,377 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:59,379 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:59,379 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:53:59,379 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:53:59,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:53:59,380 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:59,380 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:53:59,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:59,380 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:59,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:53:59,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:59,381 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:53:59,384 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:53:59,384 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:53:59,384 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:53:59,384 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:53:59,384 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:53:59,384 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:53:59,384 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:53:59,385 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. 2023-04-16 00:53:59,385 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. started... 2023-04-16 00:53:59,385 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. as already flushing 2023-04-16 00:53:59,385 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. 2023-04-16 00:53:59,385 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:53:59,385 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:53:59,385 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:59,385 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:53:59,385 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:53:59,388 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:59,388 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:59,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:59,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:53:59,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:53:59,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:53:59,389 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:59,389 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:53:59,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:53:59,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:59,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:59,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:53:59,391 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:53:59,391 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:53:59,391 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6e34fcee[Count = 0] remaining members to acquire global barrier 2023-04-16 00:53:59,391 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:59,392 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:59,392 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:59,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:59,392 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:53:59,392 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:53:59,392 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:53:59,393 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:59,394 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:53:59,395 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:53:59,395 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:59,395 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:53:59,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:59,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:53:59,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:53:59,395 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:53:59,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:53:59,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:53:59,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:59,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:59,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:53:59,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:59,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:59,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:53:59,400 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:53:59,400 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:53:59,400 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:53:59,400 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:53:59,400 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:53:59,400 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:53:59,403 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:59,403 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:59,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:59,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:53:59,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:53:59,403 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:59,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:59,403 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:53:59,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:53:59,404 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:53:59,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:53:59,404 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:59,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:59,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:59,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:53:59,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:59,406 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:59,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:59,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:53:59,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:59,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:59,412 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:59,412 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:59,412 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:59,412 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:59,412 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:59,412 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:59,412 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:59,412 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:53:59,412 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:53:59,412 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:53:59,412 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:53:59,413 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:53:59,412 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:53:59,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:53:59,413 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:53:59,413 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:53:59,413 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:53:59,414 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:53:59,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:53:59,416 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:53:59,416 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 211 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40046 deadline: 1681606499414, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:59,416 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:53:59,416 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 190 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40000 deadline: 1681606499414, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:59,424 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:53:59,425 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 207 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40044 deadline: 1681606499424, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:59,430 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:53:59,430 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39980 deadline: 1681606499425, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:59,513 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:53:59,514 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:53:59,515 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:53:59,516 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:53:59,517 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:53:59,517 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:53:59,517 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:53:59,517 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:53:59,517 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:53:59,517 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:59,517 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:59,518 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:53:59,518 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:59,518 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:53:59,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:53:59,519 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:59,519 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:53:59,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:59,519 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:59,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:53:59,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:59,520 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:53:59,520 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:53:59,520 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:53:59,520 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:53:59,520 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:53:59,520 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:53:59,520 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:53:59,520 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:53:59,520 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. 2023-04-16 00:53:59,521 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. started... 2023-04-16 00:53:59,521 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. as already flushing 2023-04-16 00:53:59,521 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. 2023-04-16 00:53:59,521 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:53:59,521 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:53:59,521 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:59,521 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:53:59,521 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:53:59,523 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:59,523 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:59,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:59,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:53:59,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:53:59,523 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:59,523 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:53:59,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:53:59,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:53:59,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:59,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:59,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:53:59,525 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:53:59,525 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@241c70c4[Count = 0] remaining members to acquire global barrier 2023-04-16 00:53:59,525 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:53:59,525 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:59,527 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:59,527 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:59,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:59,527 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:53:59,527 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:53:59,527 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:59,527 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:53:59,527 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:53:59,528 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:59,528 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:53:59,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:59,528 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:53:59,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:53:59,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:53:59,528 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:53:59,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:53:59,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:53:59,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:59,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:59,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:53:59,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:59,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:59,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:53:59,531 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:53:59,531 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:53:59,531 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:53:59,531 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:53:59,531 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:53:59,531 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:53:59,532 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:59,532 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:53:59,532 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:59,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:59,532 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:59,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:59,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:53:59,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:53:59,533 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:53:59,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:53:59,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:53:59,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:59,535 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:59,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:59,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:53:59,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:59,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:59,541 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:59,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:53:59,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:59,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:59,543 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:59,543 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:53:59,543 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:53:59,543 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:53:59,544 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:53:59,543 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:59,544 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:59,544 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:59,543 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:53:59,543 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:53:59,544 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:53:59,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:53:59,544 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:59,544 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:53:59,544 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:59,544 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:59,544 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:53:59,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:53:59,618 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:53:59,618 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 192 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40000 deadline: 1681606499617, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:59,618 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:53:59,618 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 213 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40046 deadline: 1681606499618, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:59,627 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:53:59,627 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 209 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40044 deadline: 1681606499627, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:59,633 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:53:59,633 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39980 deadline: 1681606499631, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:59,644 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:53:59,645 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:53:59,647 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:53:59,648 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:53:59,648 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:53:59,648 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:53:59,648 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:53:59,648 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:53:59,648 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:53:59,648 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:59,648 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:59,650 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:53:59,650 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:59,650 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:53:59,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:53:59,650 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:59,650 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:53:59,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:59,651 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:59,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:53:59,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:59,651 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:53:59,651 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:53:59,651 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:53:59,651 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:53:59,651 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:53:59,651 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:53:59,651 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:53:59,652 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:53:59,652 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. 2023-04-16 00:53:59,652 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. started... 2023-04-16 00:53:59,652 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. as already flushing 2023-04-16 00:53:59,652 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. 2023-04-16 00:53:59,652 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:53:59,652 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:53:59,652 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:59,652 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:53:59,652 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:53:59,659 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:59,659 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:59,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:59,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:53:59,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:53:59,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:53:59,672 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:59,672 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:53:59,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:53:59,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:59,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:59,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:53:59,674 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:53:59,674 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@cef1167[Count = 0] remaining members to acquire global barrier 2023-04-16 00:53:59,674 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:53:59,674 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:59,683 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:59,683 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:59,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:59,683 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:53:59,683 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:53:59,683 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:53:59,683 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:59,683 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:53:59,687 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:59,687 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:53:59,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:59,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:53:59,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:53:59,687 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:53:59,687 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:53:59,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:53:59,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:53:59,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:59,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:59,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:53:59,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:59,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:59,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:53:59,691 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:53:59,691 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:53:59,691 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:53:59,691 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:53:59,691 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:53:59,691 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:53:59,693 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:59,693 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:59,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:59,693 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:59,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:59,693 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:53:59,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:53:59,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:53:59,694 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:59,694 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:53:59,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:53:59,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:53:59,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:59,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:59,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:53:59,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:59,697 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=488 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/.tmp/A/7ce485f9803040dea6a7d7dff6a5a31f 2023-04-16 00:53:59,697 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:59,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:59,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:53:59,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:59,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:59,701 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:59,701 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:59,701 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:59,701 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:59,701 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:59,701 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:53:59,701 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:53:59,701 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:53:59,702 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:53:59,701 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:59,701 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:53:59,702 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:59,702 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:53:59,702 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:53:59,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:53:59,702 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:53:59,702 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:53:59,702 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:53:59,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:53:59,704 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/.tmp/A/9a3b59d8c6494766b0c439b61b2cefbb as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/A/9a3b59d8c6494766b0c439b61b2cefbb 2023-04-16 00:53:59,710 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in c9039369968c422a52a2c7db83b430ba/A of c9039369968c422a52a2c7db83b430ba into 9a3b59d8c6494766b0c439b61b2cefbb(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 00:53:59,710 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for c9039369968c422a52a2c7db83b430ba: 2023-04-16 00:53:59,710 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba., storeName=c9039369968c422a52a2c7db83b430ba/A, priority=13, startTime=1681606439217; duration=0sec 2023-04-16 00:53:59,710 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 00:53:59,802 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:53:59,804 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:53:59,806 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:53:59,807 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:53:59,807 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:53:59,807 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:53:59,808 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:53:59,808 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:53:59,808 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:53:59,808 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:59,808 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:59,809 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:59,810 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:59,810 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:53:59,810 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:53:59,810 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:53:59,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:53:59,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:59,811 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:59,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:53:59,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:59,812 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:53:59,813 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:53:59,813 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:53:59,813 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:53:59,813 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:53:59,813 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:53:59,815 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:53:59,815 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:53:59,816 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. 2023-04-16 00:53:59,816 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. started... 2023-04-16 00:53:59,816 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. as already flushing 2023-04-16 00:53:59,816 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. 2023-04-16 00:53:59,816 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:53:59,816 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:53:59,816 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:59,816 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:53:59,816 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:53:59,818 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:59,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:59,818 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:59,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:53:59,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:53:59,818 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:59,818 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:53:59,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:53:59,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:53:59,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:59,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:59,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:53:59,820 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:53:59,820 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5bea9dfb[Count = 0] remaining members to acquire global barrier 2023-04-16 00:53:59,820 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:53:59,820 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:59,822 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:59,822 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:59,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:59,823 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:53:59,823 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:53:59,823 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:53:59,823 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:59,823 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:53:59,824 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:59,824 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:53:59,824 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:53:59,824 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:53:59,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:59,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:53:59,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:53:59,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:53:59,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:53:59,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:59,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:59,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:53:59,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:59,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:59,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:53:59,829 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:53:59,829 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:53:59,829 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:53:59,829 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:53:59,829 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:53:59,829 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:53:59,832 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:59,832 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:59,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:59,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:53:59,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:53:59,832 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:53:59,832 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:59,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:59,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:53:59,833 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:53:59,833 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:59,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:59,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:53:59,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:53:59,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:59,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:59,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:59,834 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:59,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:53:59,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:59,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:59,837 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:59,837 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:59,837 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:59,837 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:59,837 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:59,837 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:53:59,837 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:59,837 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:53:59,837 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:53:59,837 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:53:59,837 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:53:59,837 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:53:59,837 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:59,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:53:59,837 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:53:59,837 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:53:59,838 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:53:59,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:53:59,920 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:53:59,921 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 215 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40046 deadline: 1681606499919, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:59,921 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:53:59,921 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 194 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40000 deadline: 1681606499920, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:59,932 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:53:59,933 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 211 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40044 deadline: 1681606499930, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:59,937 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:53:59,938 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:53:59,938 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39980 deadline: 1681606499935, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:59,938 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:53:59,940 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:53:59,941 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:53:59,941 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:53:59,941 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:53:59,941 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:53:59,941 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:53:59,941 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:53:59,942 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:59,942 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:59,944 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:53:59,944 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:59,944 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:53:59,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:53:59,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:59,945 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:59,945 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:53:59,945 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:59,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:53:59,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:59,946 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:53:59,946 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:53:59,946 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:53:59,946 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:53:59,946 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:53:59,946 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:53:59,946 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:53:59,946 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:53:59,946 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. 2023-04-16 00:53:59,946 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. started... 2023-04-16 00:53:59,947 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. as already flushing 2023-04-16 00:53:59,947 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. 2023-04-16 00:53:59,947 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:53:59,947 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:53:59,947 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:59,947 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:53:59,947 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:53:59,948 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:59,948 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:59,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:59,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:53:59,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:53:59,948 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:59,949 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:53:59,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:53:59,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:53:59,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:59,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:59,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:53:59,950 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:53:59,950 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@12f0d9e4[Count = 0] remaining members to acquire global barrier 2023-04-16 00:53:59,950 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:53:59,950 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:59,951 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:59,951 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:59,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:59,952 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:53:59,952 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:53:59,952 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:53:59,952 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:59,952 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:53:59,953 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:53:59,953 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:53:59,953 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:53:59,955 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:59,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:59,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:53:59,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:53:59,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:53:59,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:53:59,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:59,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:59,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:53:59,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:59,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:59,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:53:59,959 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:53:59,959 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:53:59,959 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:53:59,959 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:53:59,959 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:53:59,959 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:53:59,962 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:59,962 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:53:59,962 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:59,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:59,962 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:59,963 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:53:59,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:53:59,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:59,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:53:59,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:53:59,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:59,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:53:59,964 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:59,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:59,965 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:59,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:53:59,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:53:59,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:59,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:53:59,967 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:59,968 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:59,968 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:53:59,968 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:53:59,968 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:59,968 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:53:59,968 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:53:59,968 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:53:59,967 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:53:59,968 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:53:59,968 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:53:59,968 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:53:59,968 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:53:59,968 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:53:59,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:53:59,968 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:53:59,968 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:53:59,969 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:53:59,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:54:00,068 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:54:00,069 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:54:00,071 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:54:00,072 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:54:00,073 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:54:00,073 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:54:00,073 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:54:00,073 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:54:00,073 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:54:00,073 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:00,073 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:00,074 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:54:00,074 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:00,074 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:54:00,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:54:00,075 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:00,075 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:54:00,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:00,075 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:00,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:54:00,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:00,076 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:54:00,076 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:54:00,076 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:54:00,076 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:54:00,076 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:54:00,076 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:54:00,077 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:54:00,077 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:54:00,080 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. 2023-04-16 00:54:00,080 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. started... 2023-04-16 00:54:00,080 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. as already flushing 2023-04-16 00:54:00,080 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. 2023-04-16 00:54:00,080 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:54:00,080 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:54:00,080 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:00,080 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:54:00,080 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:54:00,083 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:00,083 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:00,083 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:54:00,083 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:00,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:00,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:00,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:00,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:00,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:00,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:00,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:00,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:00,087 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:54:00,087 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@55ad112e[Count = 0] remaining members to acquire global barrier 2023-04-16 00:54:00,087 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:54:00,087 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:00,088 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:00,088 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:00,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:00,088 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:54:00,088 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:54:00,088 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:54:00,089 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:00,089 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:54:00,093 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:54:00,093 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:54:00,093 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:00,093 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:54:00,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:00,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:00,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:00,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:00,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:00,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:00,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:00,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:00,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:00,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:00,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:54:00,096 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:54:00,096 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:54:00,096 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:54:00,096 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:54:00,096 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:54:00,096 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:54:00,098 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:00,098 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:54:00,098 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:00,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:00,098 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:00,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:00,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:00,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:00,098 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:54:00,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:54:00,098 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:00,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:00,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:00,099 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:00,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:00,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:00,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:00,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:00,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:00,103 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:00,103 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:00,103 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:00,103 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:00,103 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:00,103 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:00,103 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:54:00,103 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:54:00,103 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:54:00,103 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:54:00,103 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:00,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:54:00,103 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:54:00,104 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:54:00,104 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:54:00,104 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:54:00,103 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:54:00,104 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:54:00,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:54:00,114 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=488 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/.tmp/B/382d47a103644438ba02eb8cd65e655c 2023-04-16 00:54:00,204 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:54:00,205 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:54:00,207 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:54:00,208 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:54:00,208 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:54:00,208 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:54:00,208 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:54:00,208 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:54:00,208 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:54:00,208 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:00,208 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:00,210 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:00,210 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:54:00,210 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:54:00,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:54:00,211 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:00,211 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:54:00,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:00,213 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:00,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:54:00,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:00,214 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:54:00,214 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:54:00,214 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:54:00,214 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:54:00,214 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:54:00,214 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:54:00,215 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:54:00,215 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:54:00,215 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. 2023-04-16 00:54:00,215 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. started... 2023-04-16 00:54:00,215 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. as already flushing 2023-04-16 00:54:00,215 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. 2023-04-16 00:54:00,215 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:54:00,215 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:54:00,215 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:00,215 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:54:00,215 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:54:00,217 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:00,217 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:00,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:00,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:00,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:00,217 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:00,217 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:54:00,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:00,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:00,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:00,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:00,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:00,219 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:54:00,219 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@21bd46d5[Count = 0] remaining members to acquire global barrier 2023-04-16 00:54:00,219 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:54:00,219 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:00,220 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:00,220 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:00,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:00,220 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:54:00,220 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:00,220 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:54:00,220 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:54:00,221 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:54:00,222 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:00,222 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:54:00,222 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:54:00,222 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:54:00,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:00,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:00,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:00,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:00,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:00,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:00,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:00,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:00,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:00,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:00,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:54:00,230 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:54:00,230 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:54:00,230 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:54:00,230 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:54:00,230 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:54:00,230 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:54:00,238 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:00,238 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:00,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:00,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:00,239 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:00,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:00,239 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:54:00,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:00,239 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:54:00,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:54:00,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:00,239 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:00,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:00,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:00,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:00,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:00,242 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:00,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:00,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:00,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:00,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:00,246 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:00,246 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:00,246 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:54:00,246 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:54:00,246 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:54:00,246 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:54:00,246 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:54:00,246 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:54:00,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:54:00,246 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:54:00,246 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:00,246 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:00,246 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:54:00,246 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:54:00,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:54:00,246 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:54:00,246 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:00,247 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:00,247 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:00,347 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:54:00,347 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:54:00,349 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:54:00,350 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:54:00,350 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:54:00,350 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:54:00,350 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:54:00,350 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:54:00,350 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:54:00,351 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:00,351 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:00,352 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:00,352 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:54:00,352 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:54:00,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:54:00,352 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:00,352 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:54:00,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:00,353 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:00,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:54:00,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:00,354 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:54:00,354 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:54:00,354 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:54:00,354 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:54:00,354 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:54:00,354 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:54:00,364 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:54:00,364 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:54:00,364 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. 2023-04-16 00:54:00,364 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. started... 2023-04-16 00:54:00,364 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. as already flushing 2023-04-16 00:54:00,364 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. 2023-04-16 00:54:00,364 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:54:00,364 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:54:00,364 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:00,364 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:54:00,364 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:54:00,366 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:00,366 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:00,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:00,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:00,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:00,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:00,367 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:00,367 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:54:00,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:00,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:00,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:00,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:00,368 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:54:00,369 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2e37125c[Count = 0] remaining members to acquire global barrier 2023-04-16 00:54:00,369 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:54:00,369 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:00,371 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:00,371 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:00,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:00,371 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:54:00,371 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:54:00,371 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:54:00,371 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:00,371 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:54:00,372 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:00,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:00,372 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:54:00,372 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:54:00,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:00,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:00,372 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:54:00,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:00,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:00,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:00,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:00,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:00,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:00,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:00,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:54:00,377 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:54:00,377 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:54:00,378 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:54:00,378 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:54:00,378 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:54:00,378 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:54:00,380 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:00,380 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:00,380 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:00,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:00,380 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:54:00,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:00,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:00,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:00,381 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:54:00,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:54:00,381 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:00,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:00,382 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:00,384 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:00,384 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:00,384 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:00,384 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:00,384 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:00,384 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:54:00,384 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:54:00,384 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:54:00,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:54:00,384 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:00,384 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:00,385 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:54:00,385 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:54:00,385 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:54:00,385 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:54:00,385 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:54:00,385 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:54:00,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:54:00,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:00,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:00,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:00,390 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:54:00,423 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:00,423 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 217 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40046 deadline: 1681606500423, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:00,427 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:00,428 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 196 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40000 deadline: 1681606500426, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:00,441 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:00,442 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 213 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40044 deadline: 1681606500438, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:00,449 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:00,449 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39980 deadline: 1681606500446, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:00,485 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:54:00,486 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:54:00,488 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:54:00,489 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:54:00,489 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:54:00,489 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:54:00,489 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:54:00,489 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:54:00,489 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:54:00,490 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:00,490 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:00,496 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:54:00,496 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:54:00,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:54:00,496 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:00,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:00,497 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:00,497 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:00,497 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:54:00,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:54:00,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:00,499 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:54:00,499 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:54:00,499 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:54:00,499 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:54:00,499 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:54:00,500 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:54:00,501 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:54:00,501 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:54:00,501 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. 2023-04-16 00:54:00,501 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. started... 2023-04-16 00:54:00,501 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. as already flushing 2023-04-16 00:54:00,501 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. 2023-04-16 00:54:00,502 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:54:00,502 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:54:00,502 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:00,502 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:54:00,502 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:54:00,507 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:00,507 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:00,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:00,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:00,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:00,507 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:00,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:00,507 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:54:00,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:00,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:00,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:00,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:00,513 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:54:00,513 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3a9cd187[Count = 0] remaining members to acquire global barrier 2023-04-16 00:54:00,513 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:54:00,513 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:00,515 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:00,515 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:00,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:00,515 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:54:00,515 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:54:00,516 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:54:00,516 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:00,516 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:54:00,518 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:54:00,518 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:00,518 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:54:00,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:00,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:00,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:00,518 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:54:00,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:00,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:00,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:00,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:00,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:00,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:00,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:00,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:54:00,522 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:54:00,522 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:54:00,522 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:54:00,522 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:54:00,522 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:54:00,522 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:54:00,523 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:00,523 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:00,524 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:00,524 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:54:00,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:00,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:00,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:00,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:00,524 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:54:00,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:54:00,524 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:00,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:00,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:00,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:00,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:00,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:00,525 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:00,526 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=488 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/.tmp/C/648f5dd20c6749969b2473193cf7877d 2023-04-16 00:54:00,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:00,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:00,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:00,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:00,531 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/.tmp/A/7ce485f9803040dea6a7d7dff6a5a31f as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/A/7ce485f9803040dea6a7d7dff6a5a31f 2023-04-16 00:54:00,536 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:00,536 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:00,536 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:00,536 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:54:00,536 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:54:00,536 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:54:00,536 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:00,536 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:00,536 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:00,536 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:54:00,536 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:54:00,536 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:00,537 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/A/7ce485f9803040dea6a7d7dff6a5a31f, entries=200, sequenceid=488, filesize=14.3 K 2023-04-16 00:54:00,537 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/.tmp/B/382d47a103644438ba02eb8cd65e655c as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/B/382d47a103644438ba02eb8cd65e655c 2023-04-16 00:54:00,542 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:54:00,542 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:54:00,542 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:54:00,542 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:54:00,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:54:00,543 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:54:00,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:54:00,543 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/B/382d47a103644438ba02eb8cd65e655c, entries=150, sequenceid=488, filesize=11.9 K 2023-04-16 00:54:00,544 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/.tmp/C/648f5dd20c6749969b2473193cf7877d as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/C/648f5dd20c6749969b2473193cf7877d 2023-04-16 00:54:00,554 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/C/648f5dd20c6749969b2473193cf7877d, entries=150, sequenceid=488, filesize=11.9 K 2023-04-16 00:54:00,555 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~86.11 KB/88176, currentSize=154.31 KB/158010 for c9039369968c422a52a2c7db83b430ba in 1319ms, sequenceid=488, compaction requested=false 2023-04-16 00:54:00,555 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for c9039369968c422a52a2c7db83b430ba: 2023-04-16 00:54:00,636 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:54:00,637 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:54:00,655 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:54:00,656 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:54:00,657 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:54:00,657 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:54:00,657 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:54:00,657 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:54:00,657 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:54:00,657 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:00,657 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:00,658 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:00,658 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:54:00,658 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:54:00,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:54:00,659 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:00,659 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:54:00,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:00,659 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:00,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:54:00,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:00,660 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:54:00,660 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:54:00,660 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:54:00,660 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:54:00,660 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:54:00,660 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:54:00,660 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:54:00,660 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. 2023-04-16 00:54:00,660 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. started... 2023-04-16 00:54:00,660 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing c9039369968c422a52a2c7db83b430ba 3/3 column families, dataSize=154.31 KB heapSize=430.69 KB 2023-04-16 00:54:00,661 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c9039369968c422a52a2c7db83b430ba, store=A 2023-04-16 00:54:00,661 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 00:54:00,661 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c9039369968c422a52a2c7db83b430ba, store=B 2023-04-16 00:54:00,661 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 00:54:00,661 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c9039369968c422a52a2c7db83b430ba, store=C 2023-04-16 00:54:00,661 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 00:54:01,073 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=528 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/.tmp/A/ff495fbceb8f4a1eb13bb64483c6e2c6 2023-04-16 00:54:01,427 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(9158): Flush requested on c9039369968c422a52a2c7db83b430ba 2023-04-16 00:54:01,427 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. as already flushing 2023-04-16 00:54:01,434 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:01,434 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 223 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40046 deadline: 1681606501434, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:01,436 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:01,436 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 199 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40000 deadline: 1681606501434, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:01,446 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:01,446 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 215 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40044 deadline: 1681606501445, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:01,456 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:01,456 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39980 deadline: 1681606501454, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:01,488 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=528 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/.tmp/B/99d0077454834c87a9d7565881c35efa 2023-04-16 00:54:01,536 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:01,536 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 225 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40046 deadline: 1681606501536, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:01,538 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:01,538 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 201 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40000 deadline: 1681606501537, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:01,741 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:01,741 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 227 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40046 deadline: 1681606501737, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:01,741 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:01,742 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 203 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40000 deadline: 1681606501740, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:01,901 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=528 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/.tmp/C/7834ad522c7041afaefd26df423307c6 2023-04-16 00:54:01,906 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/.tmp/A/ff495fbceb8f4a1eb13bb64483c6e2c6 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/A/ff495fbceb8f4a1eb13bb64483c6e2c6 2023-04-16 00:54:01,911 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/A/ff495fbceb8f4a1eb13bb64483c6e2c6, entries=150, sequenceid=528, filesize=11.9 K 2023-04-16 00:54:01,912 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/.tmp/B/99d0077454834c87a9d7565881c35efa as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/B/99d0077454834c87a9d7565881c35efa 2023-04-16 00:54:01,916 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/B/99d0077454834c87a9d7565881c35efa, entries=150, sequenceid=528, filesize=11.9 K 2023-04-16 00:54:01,917 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/.tmp/C/7834ad522c7041afaefd26df423307c6 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/C/7834ad522c7041afaefd26df423307c6 2023-04-16 00:54:01,922 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/C/7834ad522c7041afaefd26df423307c6, entries=150, sequenceid=528, filesize=11.9 K 2023-04-16 00:54:01,923 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~154.31 KB/158010, heapSize ~430.64 KB/440976, currentSize=33.54 KB/34350 for c9039369968c422a52a2c7db83b430ba in 1263ms, sequenceid=528, compaction requested=true 2023-04-16 00:54:01,923 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for c9039369968c422a52a2c7db83b430ba: 2023-04-16 00:54:01,923 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. 2023-04-16 00:54:01,923 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:54:01,923 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:54:01,923 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:01,924 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:54:01,924 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:54:01,925 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:01,926 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:01,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:01,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:01,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:01,926 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:01,926 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:54:01,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:01,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:01,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:01,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:01,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:01,927 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:54:01,927 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@705839b[Count = 0] remaining members to acquire global barrier 2023-04-16 00:54:01,927 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:54:01,927 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:01,928 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:01,928 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:01,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:01,929 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:54:01,929 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:54:01,929 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:01,929 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:54:01,929 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:54:01,932 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:01,932 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:54:01,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:01,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:01,932 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:54:01,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:01,932 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:54:01,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:01,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:01,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:01,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:01,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:01,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:01,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:01,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:54:01,935 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:54:01,935 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:54:01,935 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:54:01,935 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:54:01,935 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:54:01,935 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:54:01,939 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:01,939 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:54:01,939 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:01,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:01,939 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:54:01,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:54:01,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:01,941 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:01,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:01,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:01,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:01,942 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:01,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:01,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:01,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:01,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:01,943 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:01,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:01,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:01,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:01,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:01,945 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:54:01,945 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:54:01,945 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:54:01,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:54:01,945 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:01,945 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:01,945 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:01,945 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:54:01,945 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:54:01,946 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:54:01,945 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:01,946 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:01,946 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:01,946 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:01,946 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:54:01,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:54:01,946 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:54:01,946 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:54:01,946 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:54:02,046 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:54:02,047 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(9158): Flush requested on c9039369968c422a52a2c7db83b430ba 2023-04-16 00:54:02,047 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:54:02,047 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing c9039369968c422a52a2c7db83b430ba 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-16 00:54:02,047 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c9039369968c422a52a2c7db83b430ba, store=A 2023-04-16 00:54:02,047 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:54:02,047 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c9039369968c422a52a2c7db83b430ba, store=B 2023-04-16 00:54:02,047 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:54:02,048 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c9039369968c422a52a2c7db83b430ba, store=C 2023-04-16 00:54:02,048 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:54:02,050 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:54:02,051 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:54:02,051 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:54:02,051 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:54:02,051 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:54:02,052 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:54:02,052 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:54:02,052 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:02,052 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:02,054 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:54:02,054 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:54:02,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:54:02,055 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:02,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:02,056 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:02,056 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:54:02,056 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:02,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:54:02,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:02,057 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:54:02,057 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:54:02,057 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:54:02,057 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:54:02,057 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:54:02,057 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:54:02,057 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:54:02,057 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:54:02,057 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. 2023-04-16 00:54:02,057 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. started... 2023-04-16 00:54:02,057 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. as already flushing 2023-04-16 00:54:02,058 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. 2023-04-16 00:54:02,058 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:54:02,058 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:54:02,058 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:02,058 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:54:02,058 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:54:02,060 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:02,060 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:02,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:02,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:02,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:02,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:02,062 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:02,062 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:54:02,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:02,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:02,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:02,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:02,067 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:54:02,067 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4cc5e640[Count = 0] remaining members to acquire global barrier 2023-04-16 00:54:02,067 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:54:02,067 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:02,069 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:02,069 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:02,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:02,069 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:54:02,069 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:54:02,069 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:54:02,069 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:02,069 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:54:02,072 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:02,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:02,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:02,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:02,072 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:54:02,072 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:54:02,072 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:54:02,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:02,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:02,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:02,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:02,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:02,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:02,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:02,079 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=538 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/.tmp/A/76f2c4cb651e41e48df4cc3fba495fb1 2023-04-16 00:54:02,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:54:02,079 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:54:02,079 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:54:02,080 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:54:02,080 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:54:02,080 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:54:02,080 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:54:02,082 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:02,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:02,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:02,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:02,083 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:02,083 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:54:02,083 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:02,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:02,083 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:54:02,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:02,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:54:02,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:02,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:02,084 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:02,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:02,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:02,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:02,087 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:02,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:02,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:02,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:02,093 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:54:02,093 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:54:02,093 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:54:02,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:54:02,096 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681606442071, max timestamp=1681606442094Num uniques 859; 2023-04-16 00:54:02,096 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-16 00:54:02,096 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:54:02,096 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1681606442071, max timestamp=1681606442094Num uniques 859; 2023-04-16 00:54:02,096 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:54:02,096 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:54:02,096 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681606442071, max timestamp=1681606442094Num uniques 859; 2023-04-16 00:54:02,096 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-16 00:54:02,096 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-16 00:54:02,096 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:54:02,097 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:54:02,097 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:54:02,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:54:02,098 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:02,098 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:02,098 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:02,098 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:02,098 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:02,098 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:02,098 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:02,130 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=538 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/.tmp/B/69a80f710d014a00b6db0ef73b5062cb 2023-04-16 00:54:02,136 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:02,136 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 221 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40000 deadline: 1681606502131, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:02,136 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:02,136 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 249 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40046 deadline: 1681606502131, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:02,157 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=538 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/.tmp/C/c9d1fa345c1844768b0413a43117c79b 2023-04-16 00:54:02,173 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/.tmp/A/76f2c4cb651e41e48df4cc3fba495fb1 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/A/76f2c4cb651e41e48df4cc3fba495fb1 2023-04-16 00:54:02,178 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/A/76f2c4cb651e41e48df4cc3fba495fb1, entries=150, sequenceid=538, filesize=11.9 K 2023-04-16 00:54:02,179 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/.tmp/B/69a80f710d014a00b6db0ef73b5062cb as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/B/69a80f710d014a00b6db0ef73b5062cb 2023-04-16 00:54:02,187 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/B/69a80f710d014a00b6db0ef73b5062cb, entries=150, sequenceid=538, filesize=11.9 K 2023-04-16 00:54:02,188 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/.tmp/C/c9d1fa345c1844768b0413a43117c79b as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/C/c9d1fa345c1844768b0413a43117c79b 2023-04-16 00:54:02,192 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/C/c9d1fa345c1844768b0413a43117c79b, entries=150, sequenceid=538, filesize=11.9 K 2023-04-16 00:54:02,193 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=140.89 KB/144270 for c9039369968c422a52a2c7db83b430ba in 146ms, sequenceid=538, compaction requested=true 2023-04-16 00:54:02,193 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for c9039369968c422a52a2c7db83b430ba: 2023-04-16 00:54:02,193 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 00:54:02,193 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-16 00:54:02,193 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 00:54:02,194 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-16 00:54:02,194 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-16 00:54:02,195 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 52138 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-16 00:54:02,195 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HStore(1912): c9039369968c422a52a2c7db83b430ba/A is initiating minor compaction (all files) 2023-04-16 00:54:02,195 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49698 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-16 00:54:02,195 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of c9039369968c422a52a2c7db83b430ba/A in TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. 2023-04-16 00:54:02,195 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HStore(1912): c9039369968c422a52a2c7db83b430ba/B is initiating minor compaction (all files) 2023-04-16 00:54:02,195 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/A/9a3b59d8c6494766b0c439b61b2cefbb, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/A/7ce485f9803040dea6a7d7dff6a5a31f, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/A/ff495fbceb8f4a1eb13bb64483c6e2c6, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/A/76f2c4cb651e41e48df4cc3fba495fb1] into tmpdir=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/.tmp, totalSize=50.9 K 2023-04-16 00:54:02,195 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HRegion(2259): Starting compaction of c9039369968c422a52a2c7db83b430ba/B in TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. 2023-04-16 00:54:02,195 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/B/54936fa0a5084f72b8636cdc748cb8af, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/B/382d47a103644438ba02eb8cd65e655c, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/B/99d0077454834c87a9d7565881c35efa, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/B/69a80f710d014a00b6db0ef73b5062cb] into tmpdir=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/.tmp, totalSize=48.5 K 2023-04-16 00:54:02,196 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.Compactor(207): Compacting 9a3b59d8c6494766b0c439b61b2cefbb, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=466, earliestPutTs=1681606439073 2023-04-16 00:54:02,196 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting 54936fa0a5084f72b8636cdc748cb8af, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=466, earliestPutTs=1681606439073 2023-04-16 00:54:02,196 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.Compactor(207): Compacting 7ce485f9803040dea6a7d7dff6a5a31f, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=488, earliestPutTs=1681606439099 2023-04-16 00:54:02,196 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting 382d47a103644438ba02eb8cd65e655c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=488, earliestPutTs=1681606439099 2023-04-16 00:54:02,197 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:54:02,197 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.Compactor(207): Compacting ff495fbceb8f4a1eb13bb64483c6e2c6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=528, earliestPutTs=1681606439293 2023-04-16 00:54:02,197 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting 99d0077454834c87a9d7565881c35efa, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=528, earliestPutTs=1681606439293 2023-04-16 00:54:02,197 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:54:02,198 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.Compactor(207): Compacting 76f2c4cb651e41e48df4cc3fba495fb1, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=538, earliestPutTs=1681606441433 2023-04-16 00:54:02,198 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting 69a80f710d014a00b6db0ef73b5062cb, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=538, earliestPutTs=1681606441433 2023-04-16 00:54:02,199 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:54:02,216 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:54:02,216 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:54:02,216 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:54:02,216 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:54:02,216 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:54:02,216 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:54:02,216 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:02,217 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:02,218 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:54:02,218 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:02,218 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:54:02,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:54:02,218 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:02,218 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:54:02,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:02,218 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:02,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:54:02,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:02,219 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:54:02,225 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] throttle.PressureAwareThroughputController(145): c9039369968c422a52a2c7db83b430ba#A#compaction#96 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-16 00:54:02,234 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] throttle.PressureAwareThroughputController(145): c9039369968c422a52a2c7db83b430ba#B#compaction#97 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-16 00:54:02,240 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:54:02,240 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:54:02,241 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:54:02,241 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:54:02,241 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:54:02,241 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:54:02,241 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:54:02,241 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. 2023-04-16 00:54:02,241 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. started... 2023-04-16 00:54:02,241 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing c9039369968c422a52a2c7db83b430ba 3/3 column families, dataSize=140.89 KB heapSize=400.92 KB 2023-04-16 00:54:02,242 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1681606442123, max timestamp=1681606442239Num uniques 1018; 2023-04-16 00:54:02,242 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-16 00:54:02,242 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1681606442123, max timestamp=1681606442239Num uniques 1168; 2023-04-16 00:54:02,242 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=53488, new segment data size=7688, suffix heap size=149440,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1168, new segment cells count=168 2023-04-16 00:54:02,242 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1681606442123, max timestamp=1681606442239Num uniques 1018; 2023-04-16 00:54:02,243 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-16 00:54:02,244 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c9039369968c422a52a2c7db83b430ba, store=A 2023-04-16 00:54:02,244 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 00:54:02,244 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c9039369968c422a52a2c7db83b430ba, store=B 2023-04-16 00:54:02,244 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 00:54:02,244 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c9039369968c422a52a2c7db83b430ba, store=C 2023-04-16 00:54:02,244 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 00:54:02,256 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(9158): Flush requested on c9039369968c422a52a2c7db83b430ba 2023-04-16 00:54:02,256 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. as already flushing 2023-04-16 00:54:02,277 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=576 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/.tmp/A/e2cd16025b6e4772b761f3981ab02028 2023-04-16 00:54:02,285 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/.tmp/B/da707cce6dc74068b739d24f59d7cfa5 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/B/da707cce6dc74068b739d24f59d7cfa5 2023-04-16 00:54:02,287 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/.tmp/A/59f005cbb1fb451f8975688d0823688f as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/A/59f005cbb1fb451f8975688d0823688f 2023-04-16 00:54:02,297 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in c9039369968c422a52a2c7db83b430ba/A of c9039369968c422a52a2c7db83b430ba into 59f005cbb1fb451f8975688d0823688f(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 00:54:02,297 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for c9039369968c422a52a2c7db83b430ba: 2023-04-16 00:54:02,297 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba., storeName=c9039369968c422a52a2c7db83b430ba/A, priority=12, startTime=1681606442193; duration=0sec 2023-04-16 00:54:02,297 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 00:54:02,297 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in c9039369968c422a52a2c7db83b430ba/B of c9039369968c422a52a2c7db83b430ba into da707cce6dc74068b739d24f59d7cfa5(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 00:54:02,297 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for c9039369968c422a52a2c7db83b430ba: 2023-04-16 00:54:02,297 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba., storeName=c9039369968c422a52a2c7db83b430ba/B, priority=12, startTime=1681606442193; duration=0sec 2023-04-16 00:54:02,297 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 00:54:02,297 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-16 00:54:02,299 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49698 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-16 00:54:02,299 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HStore(1912): c9039369968c422a52a2c7db83b430ba/C is initiating minor compaction (all files) 2023-04-16 00:54:02,300 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of c9039369968c422a52a2c7db83b430ba/C in TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. 2023-04-16 00:54:02,300 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/C/c0940893fadc4e2b8a120177cc1e09cc, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/C/648f5dd20c6749969b2473193cf7877d, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/C/7834ad522c7041afaefd26df423307c6, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/C/c9d1fa345c1844768b0413a43117c79b] into tmpdir=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/.tmp, totalSize=48.5 K 2023-04-16 00:54:02,300 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.Compactor(207): Compacting c0940893fadc4e2b8a120177cc1e09cc, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=466, earliestPutTs=1681606439073 2023-04-16 00:54:02,301 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.Compactor(207): Compacting 648f5dd20c6749969b2473193cf7877d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=488, earliestPutTs=1681606439099 2023-04-16 00:54:02,301 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.Compactor(207): Compacting 7834ad522c7041afaefd26df423307c6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=528, earliestPutTs=1681606439293 2023-04-16 00:54:02,301 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.Compactor(207): Compacting c9d1fa345c1844768b0413a43117c79b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=538, earliestPutTs=1681606441433 2023-04-16 00:54:02,307 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681606442279, max timestamp=1681606442305Num uniques 859; 2023-04-16 00:54:02,307 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-16 00:54:02,307 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681606442279, max timestamp=1681606442305Num uniques 859; 2023-04-16 00:54:02,307 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-16 00:54:02,308 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681606442279, max timestamp=1681606442305Num uniques 859; 2023-04-16 00:54:02,308 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-16 00:54:02,315 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] throttle.PressureAwareThroughputController(145): c9039369968c422a52a2c7db83b430ba#C#compaction#100 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-16 00:54:02,342 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=318, cellSize=14558, totalHeapSize=33312, min timestamp=1681606442330, max timestamp=1681606442340Num uniques 1018; 2023-04-16 00:54:02,342 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=14558, suffix heap size=133840,new segment heap size=33312 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=318 2023-04-16 00:54:02,343 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=318, cellSize=14558, totalHeapSize=33312, min timestamp=1681606442330, max timestamp=1681606442340Num uniques 1018; 2023-04-16 00:54:02,343 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=318, cellSize=14558, totalHeapSize=33312, min timestamp=1681606442330, max timestamp=1681606442340Num uniques 1018; 2023-04-16 00:54:02,343 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=14558, suffix heap size=133840,new segment heap size=33312 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=318 2023-04-16 00:54:02,343 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=14558, suffix heap size=133840,new segment heap size=33312 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=318 2023-04-16 00:54:02,373 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:02,373 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 250 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40000 deadline: 1681606502369, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:02,374 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=277, cellSize=12682, totalHeapSize=29048, min timestamp=1681606442359, max timestamp=1681606442370Num uniques 1177; 2023-04-16 00:54:02,374 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=53902, new segment data size=12682, suffix heap size=150376,new segment heap size=29048 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1177, new segment cells count=277 2023-04-16 00:54:02,377 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=277, cellSize=12682, totalHeapSize=29048, min timestamp=1681606442359, max timestamp=1681606442370Num uniques 1177; 2023-04-16 00:54:02,377 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=53902, new segment data size=12682, suffix heap size=150376,new segment heap size=29048 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1177, new segment cells count=277 2023-04-16 00:54:02,378 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=277, cellSize=12682, totalHeapSize=29048, min timestamp=1681606442359, max timestamp=1681606442370Num uniques 1177; 2023-04-16 00:54:02,378 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=53902, new segment data size=12682, suffix heap size=150376,new segment heap size=29048 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1177, new segment cells count=277 2023-04-16 00:54:02,434 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1681606442417, max timestamp=1681606442432Num uniques 1136; 2023-04-16 00:54:02,434 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=52026, new segment data size=10806, suffix heap size=146112,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1136, new segment cells count=236 2023-04-16 00:54:02,435 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1681606442417, max timestamp=1681606442432Num uniques 1136; 2023-04-16 00:54:02,435 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1681606442417, max timestamp=1681606442432Num uniques 1136; 2023-04-16 00:54:02,435 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=52026, new segment data size=10806, suffix heap size=146112,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1136, new segment cells count=236 2023-04-16 00:54:02,435 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=52026, new segment data size=10806, suffix heap size=146112,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1136, new segment cells count=236 2023-04-16 00:54:02,486 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=295, cellSize=13510, totalHeapSize=30920, min timestamp=1681606442461, max timestamp=1681606442484Num uniques 1095; 2023-04-16 00:54:02,486 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=50150, new segment data size=13510, suffix heap size=141848,new segment heap size=30920 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1095, new segment cells count=295 2023-04-16 00:54:02,487 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=295, cellSize=13510, totalHeapSize=30920, min timestamp=1681606442461, max timestamp=1681606442484Num uniques 1095; 2023-04-16 00:54:02,487 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=295, cellSize=13510, totalHeapSize=30920, min timestamp=1681606442461, max timestamp=1681606442484Num uniques 1095; 2023-04-16 00:54:02,487 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=50150, new segment data size=13510, suffix heap size=141848,new segment heap size=30920 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1095, new segment cells count=295 2023-04-16 00:54:02,487 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=50150, new segment data size=13510, suffix heap size=141848,new segment heap size=30920 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1095, new segment cells count=295 2023-04-16 00:54:02,513 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=254, cellSize=11630, totalHeapSize=26656, min timestamp=1681606442501, max timestamp=1681606442511Num uniques 1154; 2023-04-16 00:54:02,513 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=52850, new segment data size=11630, suffix heap size=147984,new segment heap size=26656 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1154, new segment cells count=254 2023-04-16 00:54:02,513 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=254, cellSize=11630, totalHeapSize=26656, min timestamp=1681606442501, max timestamp=1681606442511Num uniques 1154; 2023-04-16 00:54:02,514 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=52850, new segment data size=11630, suffix heap size=147984,new segment heap size=26656 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1154, new segment cells count=254 2023-04-16 00:54:02,514 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=254, cellSize=11630, totalHeapSize=26656, min timestamp=1681606442501, max timestamp=1681606442511Num uniques 1154; 2023-04-16 00:54:02,514 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=52850, new segment data size=11630, suffix heap size=147984,new segment heap size=26656 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1154, new segment cells count=254 2023-04-16 00:54:02,544 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=313, cellSize=14328, totalHeapSize=32792, min timestamp=1681606442526, max timestamp=1681606442542Num uniques 1113; 2023-04-16 00:54:02,544 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=50968, new segment data size=14328, suffix heap size=143720,new segment heap size=32792 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1113, new segment cells count=313 2023-04-16 00:54:02,545 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=313, cellSize=14328, totalHeapSize=32792, min timestamp=1681606442526, max timestamp=1681606442542Num uniques 1113; 2023-04-16 00:54:02,545 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=313, cellSize=14328, totalHeapSize=32792, min timestamp=1681606442526, max timestamp=1681606442542Num uniques 1113; 2023-04-16 00:54:02,545 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=50968, new segment data size=14328, suffix heap size=143720,new segment heap size=32792 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1113, new segment cells count=313 2023-04-16 00:54:02,545 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=50968, new segment data size=14328, suffix heap size=143720,new segment heap size=32792 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1113, new segment cells count=313 2023-04-16 00:54:02,575 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:02,575 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 335 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40046 deadline: 1681606502574, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:02,576 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=222, cellSize=10162, totalHeapSize=23328, min timestamp=1681606442566, max timestamp=1681606442573Num uniques 1172; 2023-04-16 00:54:02,576 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=222, cellSize=10162, totalHeapSize=23328, min timestamp=1681606442566, max timestamp=1681606442573Num uniques 1172; 2023-04-16 00:54:02,576 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=222, cellSize=10162, totalHeapSize=23328, min timestamp=1681606442566, max timestamp=1681606442573Num uniques 1172; 2023-04-16 00:54:02,576 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=53672, new segment data size=10162, suffix heap size=149856,new segment heap size=23328 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1172, new segment cells count=222 2023-04-16 00:54:02,576 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=53672, new segment data size=10162, suffix heap size=149856,new segment heap size=23328 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1172, new segment cells count=222 2023-04-16 00:54:02,576 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=53672, new segment data size=10162, suffix heap size=149856,new segment heap size=23328 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1172, new segment cells count=222 2023-04-16 00:54:02,620 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=231, cellSize=10576, totalHeapSize=24264, min timestamp=1681606442603, max timestamp=1681606442618Num uniques 1081; 2023-04-16 00:54:02,620 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=231, cellSize=10576, totalHeapSize=24264, min timestamp=1681606442603, max timestamp=1681606442618Num uniques 1081; 2023-04-16 00:54:02,620 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=49506, new segment data size=10576, suffix heap size=140392,new segment heap size=24264 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1081, new segment cells count=231 2023-04-16 00:54:02,621 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=231, cellSize=10576, totalHeapSize=24264, min timestamp=1681606442603, max timestamp=1681606442618Num uniques 1081; 2023-04-16 00:54:02,621 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=49506, new segment data size=10576, suffix heap size=140392,new segment heap size=24264 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1081, new segment cells count=231 2023-04-16 00:54:02,621 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=49506, new segment data size=10576, suffix heap size=140392,new segment heap size=24264 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1081, new segment cells count=231 2023-04-16 00:54:02,683 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=240, cellSize=10990, totalHeapSize=25200, min timestamp=1681606442673, max timestamp=1681606442681Num uniques 1090; 2023-04-16 00:54:02,683 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=49920, new segment data size=10990, suffix heap size=141328,new segment heap size=25200 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1090, new segment cells count=240 2023-04-16 00:54:02,684 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=240, cellSize=10990, totalHeapSize=25200, min timestamp=1681606442673, max timestamp=1681606442681Num uniques 1090; 2023-04-16 00:54:02,684 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=49920, new segment data size=10990, suffix heap size=141328,new segment heap size=25200 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1090, new segment cells count=240 2023-04-16 00:54:02,685 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=190, cellSize=8700, totalHeapSize=20000, min timestamp=1681606442673, max timestamp=1681606442681Num uniques 1090; 2023-04-16 00:54:02,685 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=49920, new segment data size=8700, suffix heap size=141328,new segment heap size=20000 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1090, new segment cells count=190 2023-04-16 00:54:02,716 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=249, cellSize=11404, totalHeapSize=26136, min timestamp=1681606442707, max timestamp=1681606442714Num uniques 1099; 2023-04-16 00:54:02,716 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=50334, new segment data size=11404, suffix heap size=142264,new segment heap size=26136 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1099, new segment cells count=249 2023-04-16 00:54:02,717 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=249, cellSize=11404, totalHeapSize=26136, min timestamp=1681606442707, max timestamp=1681606442714Num uniques 1099; 2023-04-16 00:54:02,717 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=50334, new segment data size=11404, suffix heap size=142264,new segment heap size=26136 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1099, new segment cells count=249 2023-04-16 00:54:02,717 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=249, cellSize=11404, totalHeapSize=26136, min timestamp=1681606442707, max timestamp=1681606442714Num uniques 1049; 2023-04-16 00:54:02,718 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=576 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/.tmp/B/3eb88dd363df448e9ec6b94bd1db85d0 2023-04-16 00:54:02,718 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=48044, new segment data size=11404, suffix heap size=137064,new segment heap size=26136 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1049, new segment cells count=249 2023-04-16 00:54:02,742 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/.tmp/C/4c108f2e62ec46eb8a0c6d025b65a751 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/C/4c108f2e62ec46eb8a0c6d025b65a751 2023-04-16 00:54:02,751 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in c9039369968c422a52a2c7db83b430ba/C of c9039369968c422a52a2c7db83b430ba into 4c108f2e62ec46eb8a0c6d025b65a751(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 00:54:02,751 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for c9039369968c422a52a2c7db83b430ba: 2023-04-16 00:54:02,751 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba., storeName=c9039369968c422a52a2c7db83b430ba/C, priority=12, startTime=1681606442194; duration=0sec 2023-04-16 00:54:02,752 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 00:54:02,756 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=208, cellSize=9520, totalHeapSize=21872, min timestamp=1681606442745, max timestamp=1681606442754Num uniques 1108; 2023-04-16 00:54:02,756 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=50740, new segment data size=9520, suffix heap size=143200,new segment heap size=21872 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1108, new segment cells count=208 2023-04-16 00:54:02,757 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=208, cellSize=9520, totalHeapSize=21872, min timestamp=1681606442745, max timestamp=1681606442754Num uniques 1108; 2023-04-16 00:54:02,757 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=50740, new segment data size=9520, suffix heap size=143200,new segment heap size=21872 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1108, new segment cells count=208 2023-04-16 00:54:02,758 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=208, cellSize=9520, totalHeapSize=21872, min timestamp=1681606442745, max timestamp=1681606442754Num uniques 1108; 2023-04-16 00:54:02,758 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=50740, new segment data size=9520, suffix heap size=143200,new segment heap size=21872 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1108, new segment cells count=208 2023-04-16 00:54:02,771 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=576 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/.tmp/C/ef328db1c5a34ce5af689ae0a510bedd 2023-04-16 00:54:02,787 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/.tmp/A/e2cd16025b6e4772b761f3981ab02028 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/A/e2cd16025b6e4772b761f3981ab02028 2023-04-16 00:54:02,793 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/A/e2cd16025b6e4772b761f3981ab02028, entries=150, sequenceid=576, filesize=11.9 K 2023-04-16 00:54:02,793 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/.tmp/B/3eb88dd363df448e9ec6b94bd1db85d0 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/B/3eb88dd363df448e9ec6b94bd1db85d0 2023-04-16 00:54:02,800 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/B/3eb88dd363df448e9ec6b94bd1db85d0, entries=150, sequenceid=576, filesize=11.9 K 2023-04-16 00:54:02,801 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/.tmp/C/ef328db1c5a34ce5af689ae0a510bedd as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/C/ef328db1c5a34ce5af689ae0a510bedd 2023-04-16 00:54:02,838 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/C/ef328db1c5a34ce5af689ae0a510bedd, entries=150, sequenceid=576, filesize=11.9 K 2023-04-16 00:54:02,840 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=167, cellSize=7642, totalHeapSize=17608, min timestamp=1681606442818, max timestamp=1681606442835Num uniques 1067; 2023-04-16 00:54:02,840 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=48862, new segment data size=7642, suffix heap size=138936,new segment heap size=17608 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1067, new segment cells count=167 2023-04-16 00:54:02,840 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=167, cellSize=7642, totalHeapSize=17608, min timestamp=1681606442818, max timestamp=1681606442837Num uniques 1067; 2023-04-16 00:54:02,840 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=48862, new segment data size=7642, suffix heap size=138936,new segment heap size=17608 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1067, new segment cells count=167 2023-04-16 00:54:02,840 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=167, cellSize=7641, totalHeapSize=17608, min timestamp=1681606442818, max timestamp=1681606442837Num uniques 1067; 2023-04-16 00:54:02,840 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=48861, new segment data size=7641, suffix heap size=138936,new segment heap size=17608 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1067, new segment cells count=167 2023-04-16 00:54:02,840 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~65.34 KB/66912, currentSize=73.80 KB/75570 for c9039369968c422a52a2c7db83b430ba in 599ms, sequenceid=576, compaction requested=false 2023-04-16 00:54:02,841 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for c9039369968c422a52a2c7db83b430ba: 2023-04-16 00:54:02,841 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. 2023-04-16 00:54:02,841 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:54:02,841 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:54:02,841 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:02,841 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:54:02,841 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:54:02,842 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:02,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:02,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:02,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:02,842 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:02,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:02,843 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:02,843 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:54:02,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:02,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:02,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:02,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:02,850 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:54:02,850 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5af6b876[Count = 0] remaining members to acquire global barrier 2023-04-16 00:54:02,850 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:54:02,850 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:02,851 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:02,851 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:02,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:02,851 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:54:02,851 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:54:02,851 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:54:02,851 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:02,851 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:54:02,853 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:54:02,853 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:54:02,853 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:02,853 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:54:02,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:02,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:02,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:02,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:02,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:02,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:02,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:02,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:02,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:02,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:02,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:54:02,856 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:54:02,856 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:54:02,856 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:54:02,856 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:54:02,856 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:54:02,856 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:54:02,859 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:02,859 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:02,859 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:02,859 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:54:02,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:02,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:02,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:02,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:02,859 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:54:02,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:54:02,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:02,859 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:02,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:02,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:02,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:02,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:02,861 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:02,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:02,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:02,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:02,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:02,867 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:54:02,867 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:54:02,867 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:54:02,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:54:02,867 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:54:02,867 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:54:02,867 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:54:02,867 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:54:02,867 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:02,867 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:02,867 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:02,868 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:02,868 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:02,868 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:02,868 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:02,868 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:54:02,868 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:54:02,868 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing c9039369968c422a52a2c7db83b430ba 3/3 column families, dataSize=53.67 KB heapSize=145.17 KB 2023-04-16 00:54:02,868 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c9039369968c422a52a2c7db83b430ba, store=A 2023-04-16 00:54:02,868 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 00:54:02,868 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c9039369968c422a52a2c7db83b430ba, store=B 2023-04-16 00:54:02,868 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 00:54:02,868 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c9039369968c422a52a2c7db83b430ba, store=C 2023-04-16 00:54:02,868 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 00:54:02,869 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(9158): Flush requested on c9039369968c422a52a2c7db83b430ba 2023-04-16 00:54:02,870 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:54:02,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:54:02,898 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=810 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/.tmp/A/53043f4788a04a55b704ebca75fc4dad 2023-04-16 00:54:02,902 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681606442890, max timestamp=1681606442899Num uniques 859; 2023-04-16 00:54:02,903 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-16 00:54:02,904 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681606442890, max timestamp=1681606442899Num uniques 859; 2023-04-16 00:54:02,904 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-16 00:54:02,904 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681606442890, max timestamp=1681606442899Num uniques 859; 2023-04-16 00:54:02,904 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-16 00:54:02,922 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=810 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/.tmp/B/73cae1998b924c16bd3cdb7ad8ce1d4d 2023-04-16 00:54:02,946 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=810 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/.tmp/C/3b547002069244838023503d420fefc6 2023-04-16 00:54:02,951 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/.tmp/A/53043f4788a04a55b704ebca75fc4dad as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/A/53043f4788a04a55b704ebca75fc4dad 2023-04-16 00:54:02,956 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/A/53043f4788a04a55b704ebca75fc4dad, entries=250, sequenceid=810, filesize=16.7 K 2023-04-16 00:54:02,957 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/.tmp/B/73cae1998b924c16bd3cdb7ad8ce1d4d as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/B/73cae1998b924c16bd3cdb7ad8ce1d4d 2023-04-16 00:54:02,963 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/B/73cae1998b924c16bd3cdb7ad8ce1d4d, entries=150, sequenceid=810, filesize=11.9 K 2023-04-16 00:54:02,964 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/.tmp/C/3b547002069244838023503d420fefc6 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/C/3b547002069244838023503d420fefc6 2023-04-16 00:54:02,968 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:54:02,969 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:54:02,970 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/C/3b547002069244838023503d420fefc6, entries=150, sequenceid=810, filesize=11.9 K 2023-04-16 00:54:02,971 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~145.13 KB/148608, currentSize=116.29 KB/119080 for c9039369968c422a52a2c7db83b430ba in 103ms, sequenceid=810, compaction requested=true 2023-04-16 00:54:02,971 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for c9039369968c422a52a2c7db83b430ba: 2023-04-16 00:54:02,971 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 00:54:02,971 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-16 00:54:02,971 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-16 00:54:02,972 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:54:02,971 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 00:54:02,972 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 00:54:02,973 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:54:02,973 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:54:02,973 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:54:02,973 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:54:02,973 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:54:02,973 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:54:02,973 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:02,973 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:02,973 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37648 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-16 00:54:02,974 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HStore(1912): c9039369968c422a52a2c7db83b430ba/B is initiating minor compaction (all files) 2023-04-16 00:54:02,974 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(9158): Flush requested on c9039369968c422a52a2c7db83b430ba 2023-04-16 00:54:02,974 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing c9039369968c422a52a2c7db83b430ba 3/3 column families, dataSize=123.00 KB heapSize=350.22 KB 2023-04-16 00:54:02,974 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 42528 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-16 00:54:02,974 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HRegion(2259): Starting compaction of c9039369968c422a52a2c7db83b430ba/B in TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. 2023-04-16 00:54:02,974 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HStore(1912): c9039369968c422a52a2c7db83b430ba/A is initiating minor compaction (all files) 2023-04-16 00:54:02,974 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c9039369968c422a52a2c7db83b430ba, store=A 2023-04-16 00:54:02,974 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of c9039369968c422a52a2c7db83b430ba/A in TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. 2023-04-16 00:54:02,974 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/B/da707cce6dc74068b739d24f59d7cfa5, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/B/3eb88dd363df448e9ec6b94bd1db85d0, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/B/73cae1998b924c16bd3cdb7ad8ce1d4d] into tmpdir=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/.tmp, totalSize=36.8 K 2023-04-16 00:54:02,974 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/A/59f005cbb1fb451f8975688d0823688f, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/A/e2cd16025b6e4772b761f3981ab02028, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/A/53043f4788a04a55b704ebca75fc4dad] into tmpdir=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/.tmp, totalSize=41.5 K 2023-04-16 00:54:02,974 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 00:54:02,974 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c9039369968c422a52a2c7db83b430ba, store=B 2023-04-16 00:54:02,974 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 00:54:02,974 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c9039369968c422a52a2c7db83b430ba, store=C 2023-04-16 00:54:02,974 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 00:54:02,975 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting da707cce6dc74068b739d24f59d7cfa5, keycount=150, bloomtype=ROW, size=13.0 K, encoding=NONE, compression=NONE, seqNum=538, earliestPutTs=1681606441433 2023-04-16 00:54:02,975 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.Compactor(207): Compacting 59f005cbb1fb451f8975688d0823688f, keycount=150, bloomtype=ROW, size=13.0 K, encoding=NONE, compression=NONE, seqNum=538, earliestPutTs=1681606441433 2023-04-16 00:54:02,975 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting 3eb88dd363df448e9ec6b94bd1db85d0, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=576, earliestPutTs=1681606442123 2023-04-16 00:54:02,975 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.Compactor(207): Compacting e2cd16025b6e4772b761f3981ab02028, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=576, earliestPutTs=1681606442123 2023-04-16 00:54:02,976 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.Compactor(207): Compacting 53043f4788a04a55b704ebca75fc4dad, keycount=250, bloomtype=ROW, size=16.7 K, encoding=NONE, compression=NONE, seqNum=810, earliestPutTs=1681606442834 2023-04-16 00:54:02,976 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting 73cae1998b924c16bd3cdb7ad8ce1d4d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=810, earliestPutTs=1681606442834 2023-04-16 00:54:02,982 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:54:02,982 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:02,982 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:54:02,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:54:02,982 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:02,982 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:54:02,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:02,983 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:02,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:54:02,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:02,984 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:54:02,984 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:54:02,984 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:54:02,984 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:54:02,984 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:54:02,984 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:54:02,985 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:54:02,985 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:54:02,985 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. 2023-04-16 00:54:02,985 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. started... 2023-04-16 00:54:02,986 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] throttle.PressureAwareThroughputController(145): c9039369968c422a52a2c7db83b430ba#A#compaction#106 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-16 00:54:02,988 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. as already flushing 2023-04-16 00:54:02,988 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. 2023-04-16 00:54:02,988 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:54:02,988 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:54:02,988 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:02,993 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] throttle.PressureAwareThroughputController(145): c9039369968c422a52a2c7db83b430ba#B#compaction#107 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-16 00:54:02,993 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:54:02,993 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:54:02,995 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:02,995 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:02,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:02,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:02,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:02,996 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:02,996 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:54:02,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:03,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:03,000 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:03,000 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 387 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40046 deadline: 1681606502996, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:03,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:03,001 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:03,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:03,001 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 361 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40000 deadline: 1681606502997, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:03,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:03,002 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:54:03,002 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6ba5a46[Count = 0] remaining members to acquire global barrier 2023-04-16 00:54:03,002 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:54:03,002 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:03,005 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:03,005 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:03,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:03,005 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:54:03,006 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:54:03,006 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:54:03,006 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:03,006 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:54:03,008 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:03,008 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:54:03,008 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:54:03,008 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:54:03,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:03,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:03,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:03,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:03,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:03,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:03,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:03,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:03,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:03,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:03,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:54:03,011 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:54:03,011 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:54:03,011 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:54:03,011 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:54:03,011 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:54:03,011 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:54:03,016 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:03,016 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:54:03,016 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:03,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:03,017 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:54:03,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:54:03,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:03,020 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:03,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:03,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:03,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:03,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:03,022 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:03,024 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:03,024 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/.tmp/B/1444ded9dd904a8e853954203a280218 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/B/1444ded9dd904a8e853954203a280218 2023-04-16 00:54:03,026 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:03,026 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:03,026 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:03,027 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:54:03,027 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:03,027 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:54:03,027 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:54:03,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:54:03,027 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:03,027 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:03,027 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:03,027 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:54:03,027 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:54:03,027 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:54:03,027 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:54:03,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:54:03,027 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:54:03,027 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:54:03,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:03,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:03,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:03,031 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:54:03,031 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in c9039369968c422a52a2c7db83b430ba/B of c9039369968c422a52a2c7db83b430ba into 1444ded9dd904a8e853954203a280218(size=13.1 K), total size for store is 13.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 00:54:03,032 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for c9039369968c422a52a2c7db83b430ba: 2023-04-16 00:54:03,032 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba., storeName=c9039369968c422a52a2c7db83b430ba/B, priority=13, startTime=1681606442971; duration=0sec 2023-04-16 00:54:03,032 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 00:54:03,032 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-16 00:54:03,034 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37648 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-16 00:54:03,034 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HStore(1912): c9039369968c422a52a2c7db83b430ba/C is initiating minor compaction (all files) 2023-04-16 00:54:03,034 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HRegion(2259): Starting compaction of c9039369968c422a52a2c7db83b430ba/C in TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. 2023-04-16 00:54:03,034 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/C/4c108f2e62ec46eb8a0c6d025b65a751, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/C/ef328db1c5a34ce5af689ae0a510bedd, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/C/3b547002069244838023503d420fefc6] into tmpdir=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/.tmp, totalSize=36.8 K 2023-04-16 00:54:03,034 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting 4c108f2e62ec46eb8a0c6d025b65a751, keycount=150, bloomtype=ROW, size=13.0 K, encoding=NONE, compression=NONE, seqNum=538, earliestPutTs=1681606441433 2023-04-16 00:54:03,035 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting ef328db1c5a34ce5af689ae0a510bedd, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=576, earliestPutTs=1681606442123 2023-04-16 00:54:03,035 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting 3b547002069244838023503d420fefc6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=810, earliestPutTs=1681606442834 2023-04-16 00:54:03,045 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] throttle.PressureAwareThroughputController(145): c9039369968c422a52a2c7db83b430ba#C#compaction#108 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-16 00:54:03,094 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:03,094 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39990 deadline: 1681606503094, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:03,097 DEBUG [Thread-149] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8185 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba., hostname=jenkins-hbase4.apache.org,34975,1681606419217, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor32.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-16 00:54:03,109 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:03,109 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:03,109 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 389 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40046 deadline: 1681606503102, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:03,110 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 363 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40000 deadline: 1681606503102, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:03,127 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:54:03,128 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:54:03,129 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:54:03,130 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:54:03,130 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:54:03,130 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:54:03,130 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:54:03,130 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:54:03,131 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:54:03,131 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:03,131 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:03,132 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:03,132 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:54:03,132 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:54:03,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:54:03,132 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:03,132 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:54:03,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:03,133 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:03,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:54:03,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:03,133 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:54:03,135 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:54:03,135 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:54:03,135 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:54:03,135 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:54:03,135 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:54:03,136 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:54:03,136 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:54:03,136 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. 2023-04-16 00:54:03,136 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. started... 2023-04-16 00:54:03,136 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. as already flushing 2023-04-16 00:54:03,136 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. 2023-04-16 00:54:03,136 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:54:03,136 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:54:03,136 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:03,136 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:54:03,136 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:54:03,138 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:03,138 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:03,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:03,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:03,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:03,138 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:03,138 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:54:03,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:03,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:03,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:03,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:03,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:03,140 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:54:03,140 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2d880851[Count = 0] remaining members to acquire global barrier 2023-04-16 00:54:03,140 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:54:03,140 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:03,141 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:03,141 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:03,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:03,141 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:54:03,141 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:54:03,141 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:54:03,141 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:03,141 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:54:03,142 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:03,142 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:54:03,142 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:54:03,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:03,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:03,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:03,142 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:54:03,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:03,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:03,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:03,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:03,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:03,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:03,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:03,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:54:03,147 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:54:03,147 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:54:03,147 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:54:03,147 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:54:03,147 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:54:03,147 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:54:03,148 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:03,148 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:54:03,148 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:03,148 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:03,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:03,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:03,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:03,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:03,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:03,149 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:54:03,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:54:03,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:03,149 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:03,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:03,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:03,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:03,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:03,150 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:03,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:03,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:03,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:03,153 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:54:03,153 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:54:03,153 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:54:03,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:54:03,154 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:54:03,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:54:03,155 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:03,155 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:03,155 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:03,155 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:03,155 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:54:03,155 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:03,155 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:54:03,155 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:54:03,155 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:03,155 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:03,155 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:54:03,155 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:54:03,255 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:54:03,256 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:54:03,259 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:54:03,268 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:54:03,268 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:54:03,268 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:54:03,268 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:54:03,268 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:54:03,268 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:54:03,268 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:03,268 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:03,269 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:03,269 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:54:03,270 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:54:03,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:54:03,270 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:03,270 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:54:03,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:03,270 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:03,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:54:03,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:03,270 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:54:03,271 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:54:03,271 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:54:03,271 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:54:03,271 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:54:03,271 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:54:03,271 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:54:03,271 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:54:03,271 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. 2023-04-16 00:54:03,271 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. started... 2023-04-16 00:54:03,271 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. as already flushing 2023-04-16 00:54:03,271 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. 2023-04-16 00:54:03,271 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:54:03,271 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:54:03,271 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:03,271 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:54:03,271 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:54:03,279 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:03,279 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:03,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:03,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:03,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:03,279 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:03,280 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:54:03,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:03,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:03,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:03,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:03,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:03,282 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:54:03,282 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4e1c0c46[Count = 0] remaining members to acquire global barrier 2023-04-16 00:54:03,282 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:54:03,282 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:03,283 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:03,283 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:03,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:03,283 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:54:03,283 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:54:03,283 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:54:03,284 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:03,284 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:54:03,286 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:03,286 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:54:03,286 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:54:03,286 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:54:03,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:03,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:03,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:03,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:03,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:03,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:03,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:03,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:03,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:03,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:03,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:54:03,289 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:54:03,289 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:54:03,289 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:54:03,289 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:54:03,289 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:54:03,289 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:54:03,291 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:03,291 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:54:03,291 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:03,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:03,292 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:54:03,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:54:03,292 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:03,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:03,292 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:03,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:03,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:03,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:03,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:03,293 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:03,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:03,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:03,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:03,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:03,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:03,295 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:03,295 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:03,295 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:03,295 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:03,295 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:03,296 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:03,296 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:03,296 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:54:03,296 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:54:03,296 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:54:03,296 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:54:03,296 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:54:03,297 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:54:03,304 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:54:03,304 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:54:03,304 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:54:03,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:54:03,306 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:54:03,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:54:03,313 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:03,313 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 365 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40000 deadline: 1681606503311, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:03,315 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:03,315 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 391 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40046 deadline: 1681606503312, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:03,397 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:54:03,397 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=845 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/.tmp/A/45634e25e23e4120be47c606e1c88f00 2023-04-16 00:54:03,397 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:54:03,399 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:54:03,401 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:54:03,401 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:54:03,401 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:54:03,401 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:54:03,401 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:54:03,401 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:54:03,402 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:03,402 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:03,404 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:54:03,404 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:03,404 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:54:03,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:54:03,404 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:03,404 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:54:03,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:03,405 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/.tmp/A/2b1b7c751e2c41c490b8c769bbd9d9f9 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/A/2b1b7c751e2c41c490b8c769bbd9d9f9 2023-04-16 00:54:03,405 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:03,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:54:03,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:03,405 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:54:03,406 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:54:03,406 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:54:03,406 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:54:03,406 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:54:03,406 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:54:03,406 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:54:03,406 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:54:03,406 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. 2023-04-16 00:54:03,406 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. started... 2023-04-16 00:54:03,406 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. as already flushing 2023-04-16 00:54:03,408 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. 2023-04-16 00:54:03,408 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:54:03,408 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:54:03,408 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:03,408 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:54:03,408 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:54:03,410 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:03,410 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:03,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:03,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:03,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:03,410 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:03,410 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:54:03,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:03,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:03,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:03,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:03,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:03,413 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:54:03,413 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@214628fb[Count = 0] remaining members to acquire global barrier 2023-04-16 00:54:03,413 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:54:03,413 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:03,414 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:03,414 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:54:03,417 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:03,418 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:03,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:03,418 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:54:03,418 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:54:03,418 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:54:03,419 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:54:03,419 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in c9039369968c422a52a2c7db83b430ba/A of c9039369968c422a52a2c7db83b430ba into 2b1b7c751e2c41c490b8c769bbd9d9f9(size=13.1 K), total size for store is 13.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 00:54:03,419 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:03,419 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for c9039369968c422a52a2c7db83b430ba: 2023-04-16 00:54:03,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:03,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:03,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:03,419 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:54:03,420 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:54:03,419 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba., storeName=c9039369968c422a52a2c7db83b430ba/A, priority=13, startTime=1681606442971; duration=0sec 2023-04-16 00:54:03,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:03,420 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 00:54:03,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:03,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:03,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:03,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:03,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:03,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:03,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:54:03,422 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:54:03,422 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:54:03,422 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:54:03,422 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:54:03,422 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:54:03,422 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:54:03,429 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:03,429 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:03,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:03,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:03,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:03,429 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:03,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:03,429 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:54:03,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:03,430 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:54:03,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:54:03,430 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:03,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:03,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:03,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:03,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:03,431 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:03,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:03,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:03,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:03,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:03,435 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:54:03,435 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:54:03,435 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:54:03,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:54:03,435 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:54:03,435 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:54:03,435 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:03,435 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:54:03,435 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:54:03,435 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:03,436 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:03,436 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:03,436 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:03,436 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:03,436 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:03,436 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:54:03,436 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:54:03,439 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:54:03,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:54:03,457 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:03,457 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 217 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40044 deadline: 1681606503454, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:03,460 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:03,460 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39980 deadline: 1681606503458, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:03,462 DEBUG [Thread-147] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4150 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba., hostname=jenkins-hbase4.apache.org,34975,1681606419217, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor32.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-16 00:54:03,465 DEBUG [Thread-141] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4151 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba., hostname=jenkins-hbase4.apache.org,34975,1681606419217, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor32.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-16 00:54:03,468 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/.tmp/C/f8394a06624343caa2bf05ef89377797 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/C/f8394a06624343caa2bf05ef89377797 2023-04-16 00:54:03,475 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in c9039369968c422a52a2c7db83b430ba/C of c9039369968c422a52a2c7db83b430ba into f8394a06624343caa2bf05ef89377797(size=13.1 K), total size for store is 13.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 00:54:03,475 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for c9039369968c422a52a2c7db83b430ba: 2023-04-16 00:54:03,475 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba., storeName=c9039369968c422a52a2c7db83b430ba/C, priority=13, startTime=1681606442972; duration=0sec 2023-04-16 00:54:03,475 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 00:54:03,536 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:54:03,537 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:54:03,541 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:54:03,542 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:54:03,542 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:54:03,542 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:54:03,542 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:54:03,542 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:54:03,542 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:54:03,542 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:03,542 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:03,565 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:54:03,565 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:03,565 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:54:03,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:54:03,565 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:03,565 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:54:03,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:03,566 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:03,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:54:03,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:03,566 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:54:03,568 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:54:03,568 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:54:03,568 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:54:03,568 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:54:03,574 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:54:03,574 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:54:03,575 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. 2023-04-16 00:54:03,575 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. started... 2023-04-16 00:54:03,575 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. as already flushing 2023-04-16 00:54:03,575 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. 2023-04-16 00:54:03,575 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:54:03,575 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:54:03,575 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:03,575 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:54:03,575 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:54:03,577 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:03,577 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:03,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:03,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:03,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:03,577 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:03,577 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:54:03,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:03,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:03,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:03,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:03,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:03,579 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:54:03,579 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:54:03,579 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@757a3eff[Count = 0] remaining members to acquire global barrier 2023-04-16 00:54:03,579 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:03,581 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:03,581 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:03,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:03,581 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:54:03,581 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:54:03,581 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:54:03,583 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:54:03,583 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:54:03,583 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:54:03,584 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:03,584 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:54:03,584 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:54:03,584 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:54:03,584 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:54:03,584 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:54:03,584 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:54:03,584 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:54:03,586 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:03,586 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:03,586 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:03,586 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:54:03,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:03,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:03,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:03,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:03,586 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:03,586 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:54:03,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:54:03,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:03,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:03,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:03,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:03,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:03,588 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:03,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:03,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:03,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:03,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:03,590 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:54:03,590 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:54:03,590 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:54:03,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:54:03,590 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:54:03,590 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:54:03,590 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:03,591 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:03,591 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:03,590 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:54:03,591 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:54:03,591 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:54:03,591 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:03,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:54:03,591 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:54:03,591 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:03,591 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:54:03,591 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:03,591 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:03,618 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:03,618 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 367 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40000 deadline: 1681606503615, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:03,618 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:03,618 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 393 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40046 deadline: 1681606503618, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:03,691 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:54:03,692 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:54:03,694 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:54:03,695 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:54:03,695 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:54:03,695 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:54:03,695 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:54:03,695 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:54:03,695 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:54:03,695 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:03,695 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:03,697 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:54:03,697 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:54:03,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:54:03,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:03,698 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:03,698 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:03,698 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:03,698 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:54:03,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:54:03,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:03,699 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:54:03,704 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:54:03,704 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:54:03,704 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:54:03,704 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:54:03,704 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:54:03,704 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:54:03,704 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:54:03,704 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. 2023-04-16 00:54:03,704 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. started... 2023-04-16 00:54:03,705 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. as already flushing 2023-04-16 00:54:03,705 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. 2023-04-16 00:54:03,705 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:54:03,705 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:54:03,705 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:03,705 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:54:03,705 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:54:03,706 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:03,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:03,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:03,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:03,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:03,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:03,707 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:03,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:03,707 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:03,707 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:54:03,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:03,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:03,708 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:54:03,708 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:54:03,708 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:03,708 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@185d6a81[Count = 0] remaining members to acquire global barrier 2023-04-16 00:54:03,709 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:03,709 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:03,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:03,709 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:54:03,709 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:54:03,709 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:54:03,709 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:03,709 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:54:03,710 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:03,710 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:54:03,710 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:54:03,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:03,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:03,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:03,711 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:54:03,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:03,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:03,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:03,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:03,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:03,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:03,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:03,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:54:03,713 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:54:03,713 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:54:03,714 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:54:03,714 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:54:03,714 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:54:03,714 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:54:03,716 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:03,716 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:03,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:03,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:03,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:03,716 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:03,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:03,716 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:54:03,717 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:54:03,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:54:03,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:03,718 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:03,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:03,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:03,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:03,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:03,719 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:03,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:03,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:03,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:03,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:03,721 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:54:03,721 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:54:03,721 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:54:03,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:54:03,722 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:54:03,722 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:54:03,722 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:54:03,722 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:54:03,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:54:03,722 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:03,722 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:54:03,722 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:54:03,722 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:54:03,722 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:03,722 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:03,722 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:03,722 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:03,722 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:03,722 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:03,822 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:54:03,823 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:54:03,825 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:54:03,826 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=845 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/.tmp/B/a6e6308eda444213a68b915b3cbe9aaa 2023-04-16 00:54:03,826 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:54:03,826 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:54:03,826 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:54:03,826 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:54:03,826 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:54:03,826 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:54:03,827 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:03,827 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:03,828 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:03,829 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:54:03,829 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:54:03,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:54:03,829 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:03,829 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:54:03,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:03,829 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:03,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:54:03,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:03,830 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:54:03,840 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:54:03,840 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:54:03,840 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:54:03,840 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:54:03,840 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:54:03,852 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:54:03,852 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:54:03,853 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. 2023-04-16 00:54:03,853 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. started... 2023-04-16 00:54:03,853 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. as already flushing 2023-04-16 00:54:03,853 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. 2023-04-16 00:54:03,853 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:54:03,853 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:54:03,853 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:03,853 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:54:03,853 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:54:03,860 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:03,860 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:03,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:03,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:03,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:03,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:03,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:03,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:03,862 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:03,862 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:54:03,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:03,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:03,863 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:54:03,863 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@360d2bf0[Count = 0] remaining members to acquire global barrier 2023-04-16 00:54:03,863 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:54:03,863 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:03,865 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:03,865 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:03,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:03,865 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:54:03,865 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:54:03,865 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:03,865 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:54:03,865 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:54:03,866 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:03,866 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:54:03,866 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:54:03,866 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:54:03,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:03,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:03,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:03,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:03,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:03,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:03,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:03,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:03,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:03,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:03,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:54:03,871 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:54:03,871 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:54:03,872 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:54:03,872 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:54:03,872 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:54:03,872 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:54:03,874 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:03,874 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:03,874 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:03,874 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:54:03,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:03,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:03,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:03,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:03,875 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:54:03,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:54:03,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:03,875 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:03,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:03,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:03,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:03,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:03,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:03,876 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:03,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:03,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:03,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:03,879 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:03,879 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:03,879 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:03,879 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:03,879 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:03,879 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:03,879 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:03,879 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:54:03,879 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:54:03,879 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:54:03,879 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:54:03,879 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:54:03,879 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:54:03,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:54:03,879 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:54:03,879 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:54:03,879 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=845 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/.tmp/C/66010489386f41e886412436ace43af5 2023-04-16 00:54:03,879 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:54:03,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:54:03,886 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/.tmp/A/45634e25e23e4120be47c606e1c88f00 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/A/45634e25e23e4120be47c606e1c88f00 2023-04-16 00:54:03,892 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/A/45634e25e23e4120be47c606e1c88f00, entries=300, sequenceid=845, filesize=19.0 K 2023-04-16 00:54:03,893 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/.tmp/B/a6e6308eda444213a68b915b3cbe9aaa as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/B/a6e6308eda444213a68b915b3cbe9aaa 2023-04-16 00:54:03,897 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/B/a6e6308eda444213a68b915b3cbe9aaa, entries=150, sequenceid=845, filesize=11.9 K 2023-04-16 00:54:03,898 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/.tmp/C/66010489386f41e886412436ace43af5 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/C/66010489386f41e886412436ace43af5 2023-04-16 00:54:03,908 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/C/66010489386f41e886412436ace43af5, entries=150, sequenceid=845, filesize=11.9 K 2023-04-16 00:54:03,909 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~123.00 KB/125950, heapSize ~350.17 KB/358576, currentSize=60.38 KB/61830 for c9039369968c422a52a2c7db83b430ba in 935ms, sequenceid=845, compaction requested=false 2023-04-16 00:54:03,909 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for c9039369968c422a52a2c7db83b430ba: 2023-04-16 00:54:03,979 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:54:03,980 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:54:03,985 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:54:03,987 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:54:03,987 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:54:03,987 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:54:03,987 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:54:03,987 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:54:03,987 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:54:03,988 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:03,988 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:03,989 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:54:03,989 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:54:03,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:54:03,989 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:03,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:03,990 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:03,990 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:03,990 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:54:03,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:54:03,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:03,991 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:54:03,991 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:54:03,991 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:54:03,991 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:54:03,991 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:54:03,991 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:54:03,991 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:54:03,991 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:54:03,991 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. 2023-04-16 00:54:03,991 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. started... 2023-04-16 00:54:03,991 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing c9039369968c422a52a2c7db83b430ba 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-16 00:54:03,992 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c9039369968c422a52a2c7db83b430ba, store=A 2023-04-16 00:54:03,992 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:54:03,992 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c9039369968c422a52a2c7db83b430ba, store=B 2023-04-16 00:54:03,992 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:54:03,992 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c9039369968c422a52a2c7db83b430ba, store=C 2023-04-16 00:54:03,992 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:54:04,001 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=860 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/.tmp/A/743a4d50b60c49c0844acb1c4a26fdd3 2023-04-16 00:54:04,027 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=860 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/.tmp/B/80e17eaba1f34ff390379c302e14291a 2023-04-16 00:54:04,073 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=860 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/.tmp/C/ab605045f2fa4eab8ab381778c36558a 2023-04-16 00:54:04,081 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/.tmp/A/743a4d50b60c49c0844acb1c4a26fdd3 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/A/743a4d50b60c49c0844acb1c4a26fdd3 2023-04-16 00:54:04,090 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/A/743a4d50b60c49c0844acb1c4a26fdd3, entries=150, sequenceid=860, filesize=11.9 K 2023-04-16 00:54:04,091 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/.tmp/B/80e17eaba1f34ff390379c302e14291a as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/B/80e17eaba1f34ff390379c302e14291a 2023-04-16 00:54:04,098 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/B/80e17eaba1f34ff390379c302e14291a, entries=150, sequenceid=860, filesize=11.9 K 2023-04-16 00:54:04,099 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/.tmp/C/ab605045f2fa4eab8ab381778c36558a as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/C/ab605045f2fa4eab8ab381778c36558a 2023-04-16 00:54:04,116 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/C/ab605045f2fa4eab8ab381778c36558a, entries=150, sequenceid=860, filesize=11.9 K 2023-04-16 00:54:04,117 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=0 B/0 for c9039369968c422a52a2c7db83b430ba in 126ms, sequenceid=860, compaction requested=true 2023-04-16 00:54:04,117 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for c9039369968c422a52a2c7db83b430ba: 2023-04-16 00:54:04,117 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. 2023-04-16 00:54:04,117 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:54:04,117 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:54:04,117 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:04,120 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:54:04,120 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:54:04,122 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:04,122 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:04,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:04,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:04,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:04,122 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:04,122 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:54:04,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:04,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:04,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:04,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:04,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:04,125 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:54:04,125 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@269cdc3a[Count = 0] remaining members to acquire global barrier 2023-04-16 00:54:04,125 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:54:04,125 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:04,127 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:04,127 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:04,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:04,128 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:54:04,128 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:54:04,128 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:04,128 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:54:04,128 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:54:04,129 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:04,129 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:54:04,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:04,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:04,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:04,129 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:54:04,130 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:54:04,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:04,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:04,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:04,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:04,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:04,134 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(9158): Flush requested on c9039369968c422a52a2c7db83b430ba 2023-04-16 00:54:04,134 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing c9039369968c422a52a2c7db83b430ba 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-16 00:54:04,135 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c9039369968c422a52a2c7db83b430ba, store=A 2023-04-16 00:54:04,135 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:54:04,135 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c9039369968c422a52a2c7db83b430ba, store=B 2023-04-16 00:54:04,135 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:54:04,135 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c9039369968c422a52a2c7db83b430ba, store=C 2023-04-16 00:54:04,135 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:54:04,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:04,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:04,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:54:04,136 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:54:04,136 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:54:04,136 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:54:04,137 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:54:04,137 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:54:04,137 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:54:04,138 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:04,138 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:54:04,138 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:04,139 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:04,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:04,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:04,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:04,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:04,139 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:54:04,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:54:04,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:04,139 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:04,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:04,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:04,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:04,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:04,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:04,141 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:04,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:04,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:04,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:04,145 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:54:04,145 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:04,145 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:54:04,145 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:04,145 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:04,145 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:54:04,145 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:54:04,145 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:54:04,145 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:54:04,145 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:04,146 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:54:04,145 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:54:04,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:54:04,146 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:54:04,146 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:04,146 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:04,146 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:04,146 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:54:04,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:54:04,197 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681606444179, max timestamp=1681606444195Num uniques 859; 2023-04-16 00:54:04,198 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681606444179, max timestamp=1681606444195Num uniques 859; 2023-04-16 00:54:04,198 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681606444179, max timestamp=1681606444195Num uniques 859; 2023-04-16 00:54:04,198 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-16 00:54:04,198 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-16 00:54:04,198 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-16 00:54:04,239 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:04,239 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 414 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40046 deadline: 1681606504236, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:04,239 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:04,239 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 391 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40000 deadline: 1681606504236, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:04,246 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:54:04,247 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:54:04,248 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:54:04,249 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:54:04,249 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:54:04,249 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:54:04,249 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:54:04,249 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:54:04,249 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:54:04,250 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:04,250 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:04,252 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:54:04,252 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:04,252 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:54:04,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:54:04,253 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:04,253 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:54:04,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:04,253 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:04,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:54:04,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:04,254 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:54:04,254 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:54:04,254 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:54:04,254 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:54:04,254 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:54:04,254 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:54:04,261 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:54:04,261 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:54:04,261 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. 2023-04-16 00:54:04,261 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. started... 2023-04-16 00:54:04,261 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. as already flushing 2023-04-16 00:54:04,261 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. 2023-04-16 00:54:04,261 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:54:04,261 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:54:04,261 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:04,261 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:54:04,261 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:54:04,263 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:04,263 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:04,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:04,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:04,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:04,263 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:04,263 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:54:04,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:04,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:04,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:04,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:04,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:04,265 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:54:04,265 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@23583d0[Count = 0] remaining members to acquire global barrier 2023-04-16 00:54:04,265 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:54:04,265 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:04,266 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:04,266 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:04,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:04,266 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:54:04,266 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:54:04,266 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:54:04,267 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:04,267 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:54:04,268 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:54:04,268 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:54:04,268 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:04,268 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:54:04,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:04,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:04,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:04,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:04,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:04,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:04,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:04,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:04,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:04,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:04,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:54:04,271 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:54:04,271 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:54:04,271 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:54:04,271 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:54:04,271 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:54:04,271 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:54:04,274 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:04,275 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:54:04,275 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:04,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:04,275 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:04,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:04,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:04,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:04,275 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:54:04,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:54:04,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:04,276 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:04,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:04,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:04,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:04,278 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:04,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:04,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:04,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:04,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:04,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:04,281 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:54:04,281 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:54:04,281 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:54:04,281 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:54:04,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:54:04,281 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:04,281 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:54:04,281 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:54:04,281 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:04,281 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:04,281 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:04,281 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:54:04,281 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:04,281 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:54:04,282 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:54:04,282 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:04,282 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:04,282 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:54:04,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:54:04,343 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:04,343 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 416 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40046 deadline: 1681606504340, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:04,343 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:04,343 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 393 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40000 deadline: 1681606504340, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:04,382 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:54:04,382 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:54:04,384 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:54:04,385 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:54:04,385 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:54:04,385 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:54:04,385 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:54:04,385 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:54:04,385 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:54:04,386 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:04,386 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:04,387 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:54:04,387 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:04,387 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:54:04,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:54:04,387 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:04,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:04,387 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:54:04,388 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:04,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:54:04,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:04,388 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:54:04,388 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:54:04,388 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:54:04,388 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:54:04,389 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:54:04,389 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:54:04,389 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:54:04,389 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:54:04,389 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. 2023-04-16 00:54:04,389 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. started... 2023-04-16 00:54:04,389 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. as already flushing 2023-04-16 00:54:04,389 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. 2023-04-16 00:54:04,389 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:54:04,389 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:54:04,389 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:04,389 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:54:04,389 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:54:04,391 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:04,391 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:04,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:04,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:04,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:04,391 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:04,391 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:54:04,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:04,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:04,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:04,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:04,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:04,393 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:54:04,393 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@71cff875[Count = 0] remaining members to acquire global barrier 2023-04-16 00:54:04,393 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:54:04,393 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:04,395 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:04,395 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:04,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:04,395 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:54:04,395 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:54:04,395 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:54:04,395 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:04,395 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:54:04,399 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:04,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:04,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:04,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:04,400 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:54:04,400 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:54:04,400 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:54:04,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:04,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:04,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:04,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:04,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:04,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:04,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:04,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:54:04,402 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:54:04,402 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:54:04,402 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:54:04,402 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:54:04,402 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:54:04,402 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:54:04,404 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:04,404 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:54:04,404 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:04,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:04,404 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:04,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:04,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:04,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:04,404 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:54:04,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:54:04,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:04,404 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:04,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:04,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:04,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:04,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:04,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:04,405 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:04,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:04,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:04,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:04,408 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:04,408 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:54:04,408 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:04,408 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:54:04,408 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:54:04,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:54:04,408 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:54:04,408 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:54:04,408 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:54:04,408 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:54:04,408 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:04,408 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:04,408 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:04,408 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:54:04,408 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:54:04,408 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:04,408 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:04,408 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:54:04,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:54:04,433 DEBUG [Thread-158] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4b02eeb3 to 127.0.0.1:53150 2023-04-16 00:54:04,433 DEBUG [Thread-158] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-16 00:54:04,434 DEBUG [Thread-154] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x04fb2b82 to 127.0.0.1:53150 2023-04-16 00:54:04,434 DEBUG [Thread-154] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-16 00:54:04,436 DEBUG [Thread-152] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1818ad0a to 127.0.0.1:53150 2023-04-16 00:54:04,436 DEBUG [Thread-152] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-16 00:54:04,441 DEBUG [Thread-156] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6129209e to 127.0.0.1:53150 2023-04-16 00:54:04,441 DEBUG [Thread-156] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-16 00:54:04,508 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:54:04,509 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:54:04,545 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:04,545 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:04,546 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 418 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40046 deadline: 1681606504545, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:04,546 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 395 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40000 deadline: 1681606504545, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:04,577 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=870 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/.tmp/A/37495720eb0f4cceb612d12c0c2ff69b 2023-04-16 00:54:04,849 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:04,849 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:04,849 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 397 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40000 deadline: 1681606504849, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:04,849 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 420 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40046 deadline: 1681606504849, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:04,988 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=870 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/.tmp/B/8fa91d2a2d9e46e694fadd89c5414480 2023-04-16 00:54:05,352 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:05,352 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:05,352 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 422 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40046 deadline: 1681606505351, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:05,352 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 399 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40000 deadline: 1681606505351, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:05,402 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=870 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/.tmp/C/06fcfe628b674f3f81787dea2de9baea 2023-04-16 00:54:05,407 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/.tmp/A/37495720eb0f4cceb612d12c0c2ff69b as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/A/37495720eb0f4cceb612d12c0c2ff69b 2023-04-16 00:54:05,411 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/A/37495720eb0f4cceb612d12c0c2ff69b, entries=100, sequenceid=870, filesize=9.5 K 2023-04-16 00:54:05,412 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/.tmp/B/8fa91d2a2d9e46e694fadd89c5414480 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/B/8fa91d2a2d9e46e694fadd89c5414480 2023-04-16 00:54:05,415 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/B/8fa91d2a2d9e46e694fadd89c5414480, entries=100, sequenceid=870, filesize=9.5 K 2023-04-16 00:54:05,416 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/.tmp/C/06fcfe628b674f3f81787dea2de9baea as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/C/06fcfe628b674f3f81787dea2de9baea 2023-04-16 00:54:05,420 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/C/06fcfe628b674f3f81787dea2de9baea, entries=100, sequenceid=870, filesize=9.5 K 2023-04-16 00:54:05,421 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=140.89 KB/144270 for c9039369968c422a52a2c7db83b430ba in 1287ms, sequenceid=870, compaction requested=true 2023-04-16 00:54:05,421 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for c9039369968c422a52a2c7db83b430ba: 2023-04-16 00:54:05,421 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 00:54:05,421 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-16 00:54:05,421 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-16 00:54:05,421 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 00:54:05,421 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 00:54:05,423 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 54812 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-16 00:54:05,423 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47492 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-16 00:54:05,423 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HStore(1912): c9039369968c422a52a2c7db83b430ba/A is initiating minor compaction (all files) 2023-04-16 00:54:05,423 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HStore(1912): c9039369968c422a52a2c7db83b430ba/B is initiating minor compaction (all files) 2023-04-16 00:54:05,423 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of c9039369968c422a52a2c7db83b430ba/A in TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. 2023-04-16 00:54:05,423 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HRegion(2259): Starting compaction of c9039369968c422a52a2c7db83b430ba/B in TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. 2023-04-16 00:54:05,423 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/A/2b1b7c751e2c41c490b8c769bbd9d9f9, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/A/45634e25e23e4120be47c606e1c88f00, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/A/743a4d50b60c49c0844acb1c4a26fdd3, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/A/37495720eb0f4cceb612d12c0c2ff69b] into tmpdir=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/.tmp, totalSize=53.5 K 2023-04-16 00:54:05,423 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/B/1444ded9dd904a8e853954203a280218, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/B/a6e6308eda444213a68b915b3cbe9aaa, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/B/80e17eaba1f34ff390379c302e14291a, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/B/8fa91d2a2d9e46e694fadd89c5414480] into tmpdir=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/.tmp, totalSize=46.4 K 2023-04-16 00:54:05,423 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.Compactor(207): Compacting 2b1b7c751e2c41c490b8c769bbd9d9f9, keycount=150, bloomtype=ROW, size=13.1 K, encoding=NONE, compression=NONE, seqNum=810, earliestPutTs=1681606442834 2023-04-16 00:54:05,423 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting 1444ded9dd904a8e853954203a280218, keycount=150, bloomtype=ROW, size=13.1 K, encoding=NONE, compression=NONE, seqNum=810, earliestPutTs=1681606442834 2023-04-16 00:54:05,424 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.Compactor(207): Compacting 45634e25e23e4120be47c606e1c88f00, keycount=300, bloomtype=ROW, size=19.0 K, encoding=NONE, compression=NONE, seqNum=845, earliestPutTs=1681606442941 2023-04-16 00:54:05,424 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting a6e6308eda444213a68b915b3cbe9aaa, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=845, earliestPutTs=1681606442944 2023-04-16 00:54:05,424 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.Compactor(207): Compacting 743a4d50b60c49c0844acb1c4a26fdd3, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=860, earliestPutTs=1681606442991 2023-04-16 00:54:05,424 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting 80e17eaba1f34ff390379c302e14291a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=860, earliestPutTs=1681606442991 2023-04-16 00:54:05,424 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.Compactor(207): Compacting 37495720eb0f4cceb612d12c0c2ff69b, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=870, earliestPutTs=1681606444132 2023-04-16 00:54:05,424 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting 8fa91d2a2d9e46e694fadd89c5414480, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=870, earliestPutTs=1681606444132 2023-04-16 00:54:05,433 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] throttle.PressureAwareThroughputController(145): c9039369968c422a52a2c7db83b430ba#A#compaction#117 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-16 00:54:05,433 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] throttle.PressureAwareThroughputController(145): c9039369968c422a52a2c7db83b430ba#B#compaction#118 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-16 00:54:05,844 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/.tmp/B/39cfcae02ea84e4abd283f061ea6cc96 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/B/39cfcae02ea84e4abd283f061ea6cc96 2023-04-16 00:54:05,844 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/.tmp/A/c1e38ed24eab4581adb7f71eb9f92d0f as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/A/c1e38ed24eab4581adb7f71eb9f92d0f 2023-04-16 00:54:05,850 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in c9039369968c422a52a2c7db83b430ba/A of c9039369968c422a52a2c7db83b430ba into c1e38ed24eab4581adb7f71eb9f92d0f(size=13.2 K), total size for store is 13.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 00:54:05,850 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in c9039369968c422a52a2c7db83b430ba/B of c9039369968c422a52a2c7db83b430ba into 39cfcae02ea84e4abd283f061ea6cc96(size=13.2 K), total size for store is 13.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 00:54:05,850 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for c9039369968c422a52a2c7db83b430ba: 2023-04-16 00:54:05,850 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for c9039369968c422a52a2c7db83b430ba: 2023-04-16 00:54:05,850 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba., storeName=c9039369968c422a52a2c7db83b430ba/A, priority=12, startTime=1681606445421; duration=0sec 2023-04-16 00:54:05,850 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba., storeName=c9039369968c422a52a2c7db83b430ba/B, priority=12, startTime=1681606445421; duration=0sec 2023-04-16 00:54:05,850 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 00:54:05,851 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-16 00:54:05,851 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 00:54:05,852 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47492 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-16 00:54:05,852 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HStore(1912): c9039369968c422a52a2c7db83b430ba/C is initiating minor compaction (all files) 2023-04-16 00:54:05,852 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of c9039369968c422a52a2c7db83b430ba/C in TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. 2023-04-16 00:54:05,852 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/C/f8394a06624343caa2bf05ef89377797, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/C/66010489386f41e886412436ace43af5, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/C/ab605045f2fa4eab8ab381778c36558a, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/C/06fcfe628b674f3f81787dea2de9baea] into tmpdir=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/.tmp, totalSize=46.4 K 2023-04-16 00:54:05,852 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.Compactor(207): Compacting f8394a06624343caa2bf05ef89377797, keycount=150, bloomtype=ROW, size=13.1 K, encoding=NONE, compression=NONE, seqNum=810, earliestPutTs=1681606442834 2023-04-16 00:54:05,853 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.Compactor(207): Compacting 66010489386f41e886412436ace43af5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=845, earliestPutTs=1681606442944 2023-04-16 00:54:05,853 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.Compactor(207): Compacting ab605045f2fa4eab8ab381778c36558a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=860, earliestPutTs=1681606442991 2023-04-16 00:54:05,854 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.Compactor(207): Compacting 06fcfe628b674f3f81787dea2de9baea, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=870, earliestPutTs=1681606444132 2023-04-16 00:54:05,862 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] throttle.PressureAwareThroughputController(145): c9039369968c422a52a2c7db83b430ba#C#compaction#119 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-16 00:54:05,871 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/.tmp/C/107040c091bf4ccf89371531d217a8c8 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/C/107040c091bf4ccf89371531d217a8c8 2023-04-16 00:54:05,876 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in c9039369968c422a52a2c7db83b430ba/C of c9039369968c422a52a2c7db83b430ba into 107040c091bf4ccf89371531d217a8c8(size=13.2 K), total size for store is 13.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 00:54:05,876 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for c9039369968c422a52a2c7db83b430ba: 2023-04-16 00:54:05,876 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba., storeName=c9039369968c422a52a2c7db83b430ba/C, priority=12, startTime=1681606445421; duration=0sec 2023-04-16 00:54:05,876 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 00:54:06,361 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(9158): Flush requested on c9039369968c422a52a2c7db83b430ba 2023-04-16 00:54:06,362 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing c9039369968c422a52a2c7db83b430ba 3/3 column families, dataSize=147.60 KB heapSize=420.84 KB 2023-04-16 00:54:06,362 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c9039369968c422a52a2c7db83b430ba, store=A 2023-04-16 00:54:06,362 DEBUG [Thread-145] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x77648431 to 127.0.0.1:53150 2023-04-16 00:54:06,362 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-16 00:54:06,362 DEBUG [Thread-145] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-16 00:54:06,362 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c9039369968c422a52a2c7db83b430ba, store=B 2023-04-16 00:54:06,362 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-16 00:54:06,362 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c9039369968c422a52a2c7db83b430ba, store=C 2023-04-16 00:54:06,362 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-16 00:54:06,363 DEBUG [Thread-143] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x0e16d41b to 127.0.0.1:53150 2023-04-16 00:54:06,363 DEBUG [Thread-143] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-16 00:54:06,771 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=911 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/.tmp/A/13b5e13d95664140a71bea03190020f9 2023-04-16 00:54:07,190 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=911 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/.tmp/B/0d7728f518a34caeb0ee8eea52dd5f42 2023-04-16 00:54:07,478 DEBUG [Thread-141] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1b886c8f to 127.0.0.1:53150 2023-04-16 00:54:07,478 DEBUG [Thread-141] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-16 00:54:07,487 DEBUG [Thread-147] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7e728705 to 127.0.0.1:53150 2023-04-16 00:54:07,487 DEBUG [Thread-147] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-16 00:54:07,601 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=911 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/.tmp/C/9f24b31541124567b0799ab0e3f49f11 2023-04-16 00:54:07,606 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/.tmp/A/13b5e13d95664140a71bea03190020f9 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/A/13b5e13d95664140a71bea03190020f9 2023-04-16 00:54:07,610 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/A/13b5e13d95664140a71bea03190020f9, entries=150, sequenceid=911, filesize=11.9 K 2023-04-16 00:54:07,611 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/.tmp/B/0d7728f518a34caeb0ee8eea52dd5f42 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/B/0d7728f518a34caeb0ee8eea52dd5f42 2023-04-16 00:54:07,615 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/B/0d7728f518a34caeb0ee8eea52dd5f42, entries=150, sequenceid=911, filesize=11.9 K 2023-04-16 00:54:07,616 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/.tmp/C/9f24b31541124567b0799ab0e3f49f11 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/C/9f24b31541124567b0799ab0e3f49f11 2023-04-16 00:54:07,621 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/C/9f24b31541124567b0799ab0e3f49f11, entries=150, sequenceid=911, filesize=11.9 K 2023-04-16 00:54:07,622 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~147.60 KB/151140, heapSize ~420.80 KB/430896, currentSize=20.13 KB/20610 for c9039369968c422a52a2c7db83b430ba in 1260ms, sequenceid=911, compaction requested=false 2023-04-16 00:54:07,622 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for c9039369968c422a52a2c7db83b430ba: 2023-04-16 00:54:13,158 DEBUG [Thread-149] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x09a575b8 to 127.0.0.1:53150 2023-04-16 00:54:13,158 DEBUG [Thread-149] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-16 00:54:13,158 INFO [Listener at localhost/33477] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-16 00:54:13,159 INFO [Listener at localhost/33477] hbase.AcidGuaranteesTestTool(394): wrote 57 2023-04-16 00:54:13,159 INFO [Listener at localhost/33477] hbase.AcidGuaranteesTestTool(394): wrote 300 2023-04-16 00:54:13,159 INFO [Listener at localhost/33477] hbase.AcidGuaranteesTestTool(394): wrote 273 2023-04-16 00:54:13,159 INFO [Listener at localhost/33477] hbase.AcidGuaranteesTestTool(394): wrote 113 2023-04-16 00:54:13,159 INFO [Listener at localhost/33477] hbase.AcidGuaranteesTestTool(394): wrote 48 2023-04-16 00:54:13,159 INFO [Listener at localhost/33477] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-16 00:54:13,159 INFO [Listener at localhost/33477] hbase.AcidGuaranteesTestTool(398): read 4108 2023-04-16 00:54:13,159 INFO [Listener at localhost/33477] hbase.AcidGuaranteesTestTool(398): read 4091 2023-04-16 00:54:13,159 INFO [Listener at localhost/33477] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-16 00:54:13,159 INFO [Listener at localhost/33477] hbase.AcidGuaranteesTestTool(402): scanned 1761 2023-04-16 00:54:13,159 INFO [Listener at localhost/33477] hbase.AcidGuaranteesTestTool(403): verified 5283 rows 2023-04-16 00:54:13,159 INFO [Listener at localhost/33477] hbase.AcidGuaranteesTestTool(402): scanned 1776 2023-04-16 00:54:13,159 INFO [Listener at localhost/33477] hbase.AcidGuaranteesTestTool(403): verified 5327 rows 2023-04-16 00:54:13,159 INFO [Listener at localhost/33477] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-16 00:54:13,159 DEBUG [Listener at localhost/33477] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2678d495 to 127.0.0.1:53150 2023-04-16 00:54:13,159 DEBUG [Listener at localhost/33477] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-16 00:54:13,164 INFO [Listener at localhost/33477] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-16 00:54:13,170 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable TestAcidGuarantees 2023-04-16 00:54:13,176 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure2.ProcedureExecutor(1029): Stored pid=12, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-16 00:54:13,179 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681606453179"}]},"ts":"1681606453179"} 2023-04-16 00:54:13,181 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-16 00:54:13,181 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-16 00:54:13,264 INFO [PEWorker-4] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-16 00:54:13,266 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=13, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=c9039369968c422a52a2c7db83b430ba, UNASSIGN}] 2023-04-16 00:54:13,268 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=13, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=c9039369968c422a52a2c7db83b430ba, UNASSIGN 2023-04-16 00:54:13,269 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=13 updating hbase:meta row=c9039369968c422a52a2c7db83b430ba, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:13,269 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681606453269"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681606453269"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681606453269"}]},"ts":"1681606453269"} 2023-04-16 00:54:13,271 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=14, ppid=13, state=RUNNABLE; CloseRegionProcedure c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217}] 2023-04-16 00:54:13,281 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-16 00:54:13,429 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close c9039369968c422a52a2c7db83b430ba 2023-04-16 00:54:13,430 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing c9039369968c422a52a2c7db83b430ba, disabling compactions & flushes 2023-04-16 00:54:13,430 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. 2023-04-16 00:54:13,430 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. 2023-04-16 00:54:13,430 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. after waiting 0 ms 2023-04-16 00:54:13,430 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. 2023-04-16 00:54:13,430 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing c9039369968c422a52a2c7db83b430ba 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-04-16 00:54:13,430 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c9039369968c422a52a2c7db83b430ba, store=A 2023-04-16 00:54:13,430 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:54:13,430 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c9039369968c422a52a2c7db83b430ba, store=B 2023-04-16 00:54:13,430 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:54:13,430 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c9039369968c422a52a2c7db83b430ba, store=C 2023-04-16 00:54:13,430 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:54:13,437 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=918 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/.tmp/A/67b53b07717a49f892f7c176c0c45cbe 2023-04-16 00:54:13,482 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-16 00:54:13,783 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-16 00:54:13,848 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=918 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/.tmp/B/2065e8b8c5924567947d4c030cfcebc1 2023-04-16 00:54:13,861 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=918 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/.tmp/C/b52dc209eb3049bda69feb5f41803903 2023-04-16 00:54:13,866 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/.tmp/A/67b53b07717a49f892f7c176c0c45cbe as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/A/67b53b07717a49f892f7c176c0c45cbe 2023-04-16 00:54:13,871 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/A/67b53b07717a49f892f7c176c0c45cbe, entries=100, sequenceid=918, filesize=9.5 K 2023-04-16 00:54:13,872 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/.tmp/B/2065e8b8c5924567947d4c030cfcebc1 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/B/2065e8b8c5924567947d4c030cfcebc1 2023-04-16 00:54:13,879 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/B/2065e8b8c5924567947d4c030cfcebc1, entries=100, sequenceid=918, filesize=9.5 K 2023-04-16 00:54:13,881 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/.tmp/C/b52dc209eb3049bda69feb5f41803903 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/C/b52dc209eb3049bda69feb5f41803903 2023-04-16 00:54:13,886 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/C/b52dc209eb3049bda69feb5f41803903, entries=100, sequenceid=918, filesize=9.5 K 2023-04-16 00:54:13,886 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=0 B/0 for c9039369968c422a52a2c7db83b430ba in 456ms, sequenceid=918, compaction requested=true 2023-04-16 00:54:13,907 DEBUG [StoreCloser-TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/A/5060fe90e1494ff1977278cbc188497d, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/A/b395a4b8fa144ace900151c416009e87, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/A/04df6097d94a4008940de9cc7c096e60, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/A/ffd4bea06da94cce9d5c37645615813f, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/A/b2f8b3bc86ca446baead5627f8710419, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/A/5cbc5ead12464029a17eec293877e462, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/A/2626362e776b4e9c9eb512a0062c34c7, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/A/9bfdf85328814f7d9f54fd8988643218, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/A/c60ca2650c964955988021e770d8da8f, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/A/8ff8ab416d904d868be9cdfec79d02e6, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/A/5a2badaf98cb4718a64362d6e8c10d3d, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/A/44badbef30794eb5842e924db40c4e1a, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/A/d1429223ed4040008faa38b9495a7a4f, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/A/4ce5db29f17d44ed8dfa68525748f796, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/A/b086ffa0bbae4374b740da259db862af, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/A/1905893bd8cf465dbb2f31993b65478d, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/A/466fae954ec74a619f64c7c59333c1d5, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/A/84134e01213642f18323474e33fe7f2d, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/A/02a8166c300b499992611a4e874179f4, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/A/1abf1d29f50d4662b0d87093ad802574, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/A/956450698f4741c890069edfacfe3aed, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/A/ebc9cf8536f84732a94acbf91ee93a31, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/A/5306221b1e9e44fb8abfe2f45f1bb09d, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/A/dcad1b35540143f6b8e7317070e485d0, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/A/9755bbfe9d7c4bac80b06b65f45845ad, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/A/ba06e0f624084caf83908a6a174a963e, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/A/69a44398ac3e4a729eb1570d447713b6, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/A/9a3b59d8c6494766b0c439b61b2cefbb, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/A/d00e67b4373d45969586af9c48ccd449, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/A/7ce485f9803040dea6a7d7dff6a5a31f, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/A/ff495fbceb8f4a1eb13bb64483c6e2c6, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/A/59f005cbb1fb451f8975688d0823688f, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/A/76f2c4cb651e41e48df4cc3fba495fb1, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/A/e2cd16025b6e4772b761f3981ab02028, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/A/53043f4788a04a55b704ebca75fc4dad, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/A/2b1b7c751e2c41c490b8c769bbd9d9f9, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/A/45634e25e23e4120be47c606e1c88f00, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/A/743a4d50b60c49c0844acb1c4a26fdd3, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/A/37495720eb0f4cceb612d12c0c2ff69b] to archive 2023-04-16 00:54:13,908 DEBUG [StoreCloser-TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-16 00:54:13,913 DEBUG [StoreCloser-TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/A/5060fe90e1494ff1977278cbc188497d to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/A/5060fe90e1494ff1977278cbc188497d 2023-04-16 00:54:13,914 DEBUG [StoreCloser-TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/A/b395a4b8fa144ace900151c416009e87 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/A/b395a4b8fa144ace900151c416009e87 2023-04-16 00:54:13,915 DEBUG [StoreCloser-TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/A/04df6097d94a4008940de9cc7c096e60 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/A/04df6097d94a4008940de9cc7c096e60 2023-04-16 00:54:13,917 DEBUG [StoreCloser-TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/A/ffd4bea06da94cce9d5c37645615813f to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/A/ffd4bea06da94cce9d5c37645615813f 2023-04-16 00:54:13,918 DEBUG [StoreCloser-TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/A/b2f8b3bc86ca446baead5627f8710419 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/A/b2f8b3bc86ca446baead5627f8710419 2023-04-16 00:54:13,919 DEBUG [StoreCloser-TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/A/5cbc5ead12464029a17eec293877e462 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/A/5cbc5ead12464029a17eec293877e462 2023-04-16 00:54:13,920 DEBUG [StoreCloser-TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/A/2626362e776b4e9c9eb512a0062c34c7 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/A/2626362e776b4e9c9eb512a0062c34c7 2023-04-16 00:54:13,922 DEBUG [StoreCloser-TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/A/9bfdf85328814f7d9f54fd8988643218 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/A/9bfdf85328814f7d9f54fd8988643218 2023-04-16 00:54:13,923 DEBUG [StoreCloser-TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/A/c60ca2650c964955988021e770d8da8f to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/A/c60ca2650c964955988021e770d8da8f 2023-04-16 00:54:13,924 DEBUG [StoreCloser-TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/A/8ff8ab416d904d868be9cdfec79d02e6 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/A/8ff8ab416d904d868be9cdfec79d02e6 2023-04-16 00:54:13,925 DEBUG [StoreCloser-TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/A/5a2badaf98cb4718a64362d6e8c10d3d to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/A/5a2badaf98cb4718a64362d6e8c10d3d 2023-04-16 00:54:13,925 DEBUG [StoreCloser-TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/A/44badbef30794eb5842e924db40c4e1a to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/A/44badbef30794eb5842e924db40c4e1a 2023-04-16 00:54:13,926 DEBUG [StoreCloser-TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/A/d1429223ed4040008faa38b9495a7a4f to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/A/d1429223ed4040008faa38b9495a7a4f 2023-04-16 00:54:13,928 DEBUG [StoreCloser-TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/A/4ce5db29f17d44ed8dfa68525748f796 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/A/4ce5db29f17d44ed8dfa68525748f796 2023-04-16 00:54:13,929 DEBUG [StoreCloser-TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/A/b086ffa0bbae4374b740da259db862af to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/A/b086ffa0bbae4374b740da259db862af 2023-04-16 00:54:13,930 DEBUG [StoreCloser-TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/A/1905893bd8cf465dbb2f31993b65478d to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/A/1905893bd8cf465dbb2f31993b65478d 2023-04-16 00:54:13,932 DEBUG [StoreCloser-TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/A/466fae954ec74a619f64c7c59333c1d5 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/A/466fae954ec74a619f64c7c59333c1d5 2023-04-16 00:54:13,933 DEBUG [StoreCloser-TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/A/84134e01213642f18323474e33fe7f2d to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/A/84134e01213642f18323474e33fe7f2d 2023-04-16 00:54:13,934 DEBUG [StoreCloser-TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/A/02a8166c300b499992611a4e874179f4 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/A/02a8166c300b499992611a4e874179f4 2023-04-16 00:54:13,935 DEBUG [StoreCloser-TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/A/1abf1d29f50d4662b0d87093ad802574 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/A/1abf1d29f50d4662b0d87093ad802574 2023-04-16 00:54:13,937 DEBUG [StoreCloser-TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/A/956450698f4741c890069edfacfe3aed to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/A/956450698f4741c890069edfacfe3aed 2023-04-16 00:54:13,938 DEBUG [StoreCloser-TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/A/ebc9cf8536f84732a94acbf91ee93a31 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/A/ebc9cf8536f84732a94acbf91ee93a31 2023-04-16 00:54:13,939 DEBUG [StoreCloser-TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/A/5306221b1e9e44fb8abfe2f45f1bb09d to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/A/5306221b1e9e44fb8abfe2f45f1bb09d 2023-04-16 00:54:13,940 DEBUG [StoreCloser-TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/A/dcad1b35540143f6b8e7317070e485d0 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/A/dcad1b35540143f6b8e7317070e485d0 2023-04-16 00:54:13,941 DEBUG [StoreCloser-TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/A/9755bbfe9d7c4bac80b06b65f45845ad to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/A/9755bbfe9d7c4bac80b06b65f45845ad 2023-04-16 00:54:13,942 DEBUG [StoreCloser-TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/A/ba06e0f624084caf83908a6a174a963e to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/A/ba06e0f624084caf83908a6a174a963e 2023-04-16 00:54:13,943 DEBUG [StoreCloser-TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/A/69a44398ac3e4a729eb1570d447713b6 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/A/69a44398ac3e4a729eb1570d447713b6 2023-04-16 00:54:13,944 DEBUG [StoreCloser-TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/A/9a3b59d8c6494766b0c439b61b2cefbb to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/A/9a3b59d8c6494766b0c439b61b2cefbb 2023-04-16 00:54:13,945 DEBUG [StoreCloser-TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/A/d00e67b4373d45969586af9c48ccd449 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/A/d00e67b4373d45969586af9c48ccd449 2023-04-16 00:54:13,946 DEBUG [StoreCloser-TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/A/7ce485f9803040dea6a7d7dff6a5a31f to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/A/7ce485f9803040dea6a7d7dff6a5a31f 2023-04-16 00:54:13,948 DEBUG [StoreCloser-TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/A/ff495fbceb8f4a1eb13bb64483c6e2c6 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/A/ff495fbceb8f4a1eb13bb64483c6e2c6 2023-04-16 00:54:13,949 DEBUG [StoreCloser-TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/A/59f005cbb1fb451f8975688d0823688f to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/A/59f005cbb1fb451f8975688d0823688f 2023-04-16 00:54:13,950 DEBUG [StoreCloser-TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/A/76f2c4cb651e41e48df4cc3fba495fb1 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/A/76f2c4cb651e41e48df4cc3fba495fb1 2023-04-16 00:54:13,951 DEBUG [StoreCloser-TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/A/e2cd16025b6e4772b761f3981ab02028 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/A/e2cd16025b6e4772b761f3981ab02028 2023-04-16 00:54:13,952 DEBUG [StoreCloser-TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/A/53043f4788a04a55b704ebca75fc4dad to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/A/53043f4788a04a55b704ebca75fc4dad 2023-04-16 00:54:13,953 DEBUG [StoreCloser-TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/A/2b1b7c751e2c41c490b8c769bbd9d9f9 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/A/2b1b7c751e2c41c490b8c769bbd9d9f9 2023-04-16 00:54:13,955 DEBUG [StoreCloser-TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/A/45634e25e23e4120be47c606e1c88f00 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/A/45634e25e23e4120be47c606e1c88f00 2023-04-16 00:54:13,956 DEBUG [StoreCloser-TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/A/743a4d50b60c49c0844acb1c4a26fdd3 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/A/743a4d50b60c49c0844acb1c4a26fdd3 2023-04-16 00:54:13,957 DEBUG [StoreCloser-TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/A/37495720eb0f4cceb612d12c0c2ff69b to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/A/37495720eb0f4cceb612d12c0c2ff69b 2023-04-16 00:54:13,980 DEBUG [StoreCloser-TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/B/bf4078543a914d21bb8934742f1d599a, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/B/430794e955ce43c284188aef09954da9, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/B/82192c04cd374d1c8c0ef3b1d200daeb, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/B/6c92c274cda945e9adc21f14b3b3c391, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/B/03fc509a50fd486a9e529b8686d7f030, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/B/b1725dab9e3e485d819a349709860508, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/B/b9d2f4ed4a644dcaa49ae82bb4bb1d6c, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/B/fd2f71d62d174dc1867610af432cfd3c, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/B/376f37c0b97947a1aabe342a7d2e924d, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/B/eb864ce94bb3479292ae3da5dc0d297d, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/B/40e5aebebbca4ac6b6b21b87079ed6e9, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/B/545db2cd89a34c40835683ea6a86a263, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/B/66dcf25f2dec4a07aaa142f3e50680b3, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/B/43f42b9fb9624279850656e3c66e092d, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/B/b296aedb2f3e4ede859bcdc7d27ee6a0, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/B/0a9e1b474e824a118b82cfe404453cbb, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/B/0598e1459d21477abb94ec7af539950c, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/B/93b265976d5e4ceeb91c4225461306dc, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/B/06ffbabf500c4a3cb0393f5878f775d1, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/B/e8cb49583cd948629f43a90a88961c17, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/B/9a5fce57a03746c4910f4096856433c3, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/B/2be077d7177e46a69ecdb89d78f53876, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/B/a5d40af55a374c5cb2763ce0ba9afb42, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/B/bf0ec8695516431c893540f38a978dcc, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/B/bb7b26c820ee462182b077a56191b08d, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/B/4fc9287e9d5d4d1795c97bbff98cd5de, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/B/ff34a3c646df407f8add805a5f987768, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/B/54936fa0a5084f72b8636cdc748cb8af, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/B/4fa4401a18944b888a1d4402241493b3, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/B/382d47a103644438ba02eb8cd65e655c, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/B/99d0077454834c87a9d7565881c35efa, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/B/da707cce6dc74068b739d24f59d7cfa5, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/B/69a80f710d014a00b6db0ef73b5062cb, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/B/3eb88dd363df448e9ec6b94bd1db85d0, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/B/1444ded9dd904a8e853954203a280218, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/B/73cae1998b924c16bd3cdb7ad8ce1d4d, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/B/a6e6308eda444213a68b915b3cbe9aaa, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/B/80e17eaba1f34ff390379c302e14291a, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/B/8fa91d2a2d9e46e694fadd89c5414480] to archive 2023-04-16 00:54:13,981 DEBUG [StoreCloser-TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-16 00:54:13,982 DEBUG [StoreCloser-TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/B/bf4078543a914d21bb8934742f1d599a to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/B/bf4078543a914d21bb8934742f1d599a 2023-04-16 00:54:13,983 DEBUG [StoreCloser-TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/B/430794e955ce43c284188aef09954da9 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/B/430794e955ce43c284188aef09954da9 2023-04-16 00:54:13,984 DEBUG [StoreCloser-TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/B/82192c04cd374d1c8c0ef3b1d200daeb to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/B/82192c04cd374d1c8c0ef3b1d200daeb 2023-04-16 00:54:13,985 DEBUG [StoreCloser-TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/B/6c92c274cda945e9adc21f14b3b3c391 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/B/6c92c274cda945e9adc21f14b3b3c391 2023-04-16 00:54:13,986 DEBUG [StoreCloser-TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/B/03fc509a50fd486a9e529b8686d7f030 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/B/03fc509a50fd486a9e529b8686d7f030 2023-04-16 00:54:13,988 DEBUG [StoreCloser-TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/B/b1725dab9e3e485d819a349709860508 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/B/b1725dab9e3e485d819a349709860508 2023-04-16 00:54:13,989 DEBUG [StoreCloser-TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/B/b9d2f4ed4a644dcaa49ae82bb4bb1d6c to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/B/b9d2f4ed4a644dcaa49ae82bb4bb1d6c 2023-04-16 00:54:13,990 DEBUG [StoreCloser-TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/B/fd2f71d62d174dc1867610af432cfd3c to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/B/fd2f71d62d174dc1867610af432cfd3c 2023-04-16 00:54:13,991 DEBUG [StoreCloser-TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/B/376f37c0b97947a1aabe342a7d2e924d to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/B/376f37c0b97947a1aabe342a7d2e924d 2023-04-16 00:54:13,992 DEBUG [StoreCloser-TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/B/eb864ce94bb3479292ae3da5dc0d297d to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/B/eb864ce94bb3479292ae3da5dc0d297d 2023-04-16 00:54:13,993 DEBUG [StoreCloser-TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/B/40e5aebebbca4ac6b6b21b87079ed6e9 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/B/40e5aebebbca4ac6b6b21b87079ed6e9 2023-04-16 00:54:13,995 DEBUG [StoreCloser-TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/B/545db2cd89a34c40835683ea6a86a263 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/B/545db2cd89a34c40835683ea6a86a263 2023-04-16 00:54:13,996 DEBUG [StoreCloser-TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/B/66dcf25f2dec4a07aaa142f3e50680b3 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/B/66dcf25f2dec4a07aaa142f3e50680b3 2023-04-16 00:54:13,997 DEBUG [StoreCloser-TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/B/43f42b9fb9624279850656e3c66e092d to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/B/43f42b9fb9624279850656e3c66e092d 2023-04-16 00:54:13,998 DEBUG [StoreCloser-TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/B/b296aedb2f3e4ede859bcdc7d27ee6a0 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/B/b296aedb2f3e4ede859bcdc7d27ee6a0 2023-04-16 00:54:13,999 DEBUG [StoreCloser-TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/B/0a9e1b474e824a118b82cfe404453cbb to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/B/0a9e1b474e824a118b82cfe404453cbb 2023-04-16 00:54:14,000 DEBUG [StoreCloser-TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/B/0598e1459d21477abb94ec7af539950c to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/B/0598e1459d21477abb94ec7af539950c 2023-04-16 00:54:14,001 DEBUG [StoreCloser-TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/B/93b265976d5e4ceeb91c4225461306dc to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/B/93b265976d5e4ceeb91c4225461306dc 2023-04-16 00:54:14,002 DEBUG [StoreCloser-TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/B/06ffbabf500c4a3cb0393f5878f775d1 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/B/06ffbabf500c4a3cb0393f5878f775d1 2023-04-16 00:54:14,003 DEBUG [StoreCloser-TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/B/e8cb49583cd948629f43a90a88961c17 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/B/e8cb49583cd948629f43a90a88961c17 2023-04-16 00:54:14,004 DEBUG [StoreCloser-TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/B/9a5fce57a03746c4910f4096856433c3 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/B/9a5fce57a03746c4910f4096856433c3 2023-04-16 00:54:14,005 DEBUG [StoreCloser-TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/B/2be077d7177e46a69ecdb89d78f53876 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/B/2be077d7177e46a69ecdb89d78f53876 2023-04-16 00:54:14,006 DEBUG [StoreCloser-TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/B/a5d40af55a374c5cb2763ce0ba9afb42 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/B/a5d40af55a374c5cb2763ce0ba9afb42 2023-04-16 00:54:14,007 DEBUG [StoreCloser-TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/B/bf0ec8695516431c893540f38a978dcc to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/B/bf0ec8695516431c893540f38a978dcc 2023-04-16 00:54:14,008 DEBUG [StoreCloser-TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/B/bb7b26c820ee462182b077a56191b08d to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/B/bb7b26c820ee462182b077a56191b08d 2023-04-16 00:54:14,009 DEBUG [StoreCloser-TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/B/4fc9287e9d5d4d1795c97bbff98cd5de to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/B/4fc9287e9d5d4d1795c97bbff98cd5de 2023-04-16 00:54:14,010 DEBUG [StoreCloser-TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/B/ff34a3c646df407f8add805a5f987768 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/B/ff34a3c646df407f8add805a5f987768 2023-04-16 00:54:14,011 DEBUG [StoreCloser-TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/B/54936fa0a5084f72b8636cdc748cb8af to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/B/54936fa0a5084f72b8636cdc748cb8af 2023-04-16 00:54:14,012 DEBUG [StoreCloser-TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/B/4fa4401a18944b888a1d4402241493b3 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/B/4fa4401a18944b888a1d4402241493b3 2023-04-16 00:54:14,013 DEBUG [StoreCloser-TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/B/382d47a103644438ba02eb8cd65e655c to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/B/382d47a103644438ba02eb8cd65e655c 2023-04-16 00:54:14,014 DEBUG [StoreCloser-TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/B/99d0077454834c87a9d7565881c35efa to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/B/99d0077454834c87a9d7565881c35efa 2023-04-16 00:54:14,015 DEBUG [StoreCloser-TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/B/da707cce6dc74068b739d24f59d7cfa5 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/B/da707cce6dc74068b739d24f59d7cfa5 2023-04-16 00:54:14,016 DEBUG [StoreCloser-TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/B/69a80f710d014a00b6db0ef73b5062cb to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/B/69a80f710d014a00b6db0ef73b5062cb 2023-04-16 00:54:14,017 DEBUG [StoreCloser-TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/B/3eb88dd363df448e9ec6b94bd1db85d0 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/B/3eb88dd363df448e9ec6b94bd1db85d0 2023-04-16 00:54:14,018 DEBUG [StoreCloser-TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/B/1444ded9dd904a8e853954203a280218 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/B/1444ded9dd904a8e853954203a280218 2023-04-16 00:54:14,019 DEBUG [StoreCloser-TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/B/73cae1998b924c16bd3cdb7ad8ce1d4d to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/B/73cae1998b924c16bd3cdb7ad8ce1d4d 2023-04-16 00:54:14,020 DEBUG [StoreCloser-TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/B/a6e6308eda444213a68b915b3cbe9aaa to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/B/a6e6308eda444213a68b915b3cbe9aaa 2023-04-16 00:54:14,021 DEBUG [StoreCloser-TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/B/80e17eaba1f34ff390379c302e14291a to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/B/80e17eaba1f34ff390379c302e14291a 2023-04-16 00:54:14,022 DEBUG [StoreCloser-TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/B/8fa91d2a2d9e46e694fadd89c5414480 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/B/8fa91d2a2d9e46e694fadd89c5414480 2023-04-16 00:54:14,025 DEBUG [StoreCloser-TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/C/0da3dfc269354e279ba38e2eed98439d, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/C/cca93818a0814e2b8ad85e1628228b7b, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/C/d68816e41a76482f91678e90e7f7411b, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/C/6ed04a0f0ac149249b0e67dbe6ef4bdd, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/C/2b2651e29329451a84a5318c553ae69c, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/C/fd0d64e0bc054fa19000e0649788fe0f, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/C/e7ef5613df4745038a32473837327c7b, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/C/11757a04a9cf417787c1a36a31497de0, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/C/d67b46721e6f4698881fd6d66dde1c0b, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/C/2564d901e187492689e4c014f1735db1, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/C/6a682d3996474f509ae80c126aebdc7a, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/C/cdb2e022925e4317841f458eb47f2f92, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/C/1d73dbf886d449af8949899e62596d47, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/C/c1e8f2ef7e1f4ec0895c7a1b6b1bebd7, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/C/08db334e63124a019778864bf984d119, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/C/ed3ec1d5ae5e4da5a297370347b5db2b, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/C/048858e9a71e4b60a0689027894b6451, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/C/e1a2b1285d9c41df841a226d149eaf57, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/C/22ae222c4480457d84e9bef4e411489e, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/C/4f0d2eafb2174fa495279f45b84c02d5, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/C/5d8b41609f4e42a18e4d23e8c688880c, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/C/68fe0c007b144c26af83d9c2ef895eec, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/C/5cda4afcb1f8485ab1ef5c121276c90c, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/C/79bfe09a6a5f49d5a49988b3191a255f, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/C/afb3b38579dc4f0fb532aa79bf8ca79c, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/C/c2040e6802c44108be891184e85171ed, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/C/64e168accb044b7aa27fa8331d3872fb, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/C/c0940893fadc4e2b8a120177cc1e09cc, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/C/a4d87411c83443929fdbe9b819ec40a4, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/C/648f5dd20c6749969b2473193cf7877d, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/C/7834ad522c7041afaefd26df423307c6, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/C/4c108f2e62ec46eb8a0c6d025b65a751, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/C/c9d1fa345c1844768b0413a43117c79b, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/C/ef328db1c5a34ce5af689ae0a510bedd, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/C/f8394a06624343caa2bf05ef89377797, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/C/3b547002069244838023503d420fefc6, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/C/66010489386f41e886412436ace43af5, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/C/ab605045f2fa4eab8ab381778c36558a, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/C/06fcfe628b674f3f81787dea2de9baea] to archive 2023-04-16 00:54:14,026 DEBUG [StoreCloser-TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-16 00:54:14,028 DEBUG [StoreCloser-TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/C/0da3dfc269354e279ba38e2eed98439d to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/C/0da3dfc269354e279ba38e2eed98439d 2023-04-16 00:54:14,029 DEBUG [StoreCloser-TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/C/cca93818a0814e2b8ad85e1628228b7b to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/C/cca93818a0814e2b8ad85e1628228b7b 2023-04-16 00:54:14,030 DEBUG [StoreCloser-TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/C/d68816e41a76482f91678e90e7f7411b to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/C/d68816e41a76482f91678e90e7f7411b 2023-04-16 00:54:14,031 DEBUG [StoreCloser-TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/C/6ed04a0f0ac149249b0e67dbe6ef4bdd to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/C/6ed04a0f0ac149249b0e67dbe6ef4bdd 2023-04-16 00:54:14,032 DEBUG [StoreCloser-TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/C/2b2651e29329451a84a5318c553ae69c to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/C/2b2651e29329451a84a5318c553ae69c 2023-04-16 00:54:14,034 DEBUG [StoreCloser-TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/C/fd0d64e0bc054fa19000e0649788fe0f to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/C/fd0d64e0bc054fa19000e0649788fe0f 2023-04-16 00:54:14,035 DEBUG [StoreCloser-TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/C/e7ef5613df4745038a32473837327c7b to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/C/e7ef5613df4745038a32473837327c7b 2023-04-16 00:54:14,036 DEBUG [StoreCloser-TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/C/11757a04a9cf417787c1a36a31497de0 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/C/11757a04a9cf417787c1a36a31497de0 2023-04-16 00:54:14,037 DEBUG [StoreCloser-TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/C/d67b46721e6f4698881fd6d66dde1c0b to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/C/d67b46721e6f4698881fd6d66dde1c0b 2023-04-16 00:54:14,038 DEBUG [StoreCloser-TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/C/2564d901e187492689e4c014f1735db1 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/C/2564d901e187492689e4c014f1735db1 2023-04-16 00:54:14,040 DEBUG [StoreCloser-TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/C/6a682d3996474f509ae80c126aebdc7a to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/C/6a682d3996474f509ae80c126aebdc7a 2023-04-16 00:54:14,041 DEBUG [StoreCloser-TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/C/cdb2e022925e4317841f458eb47f2f92 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/C/cdb2e022925e4317841f458eb47f2f92 2023-04-16 00:54:14,043 DEBUG [StoreCloser-TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/C/1d73dbf886d449af8949899e62596d47 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/C/1d73dbf886d449af8949899e62596d47 2023-04-16 00:54:14,044 DEBUG [StoreCloser-TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/C/c1e8f2ef7e1f4ec0895c7a1b6b1bebd7 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/C/c1e8f2ef7e1f4ec0895c7a1b6b1bebd7 2023-04-16 00:54:14,045 DEBUG [StoreCloser-TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/C/08db334e63124a019778864bf984d119 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/C/08db334e63124a019778864bf984d119 2023-04-16 00:54:14,046 DEBUG [StoreCloser-TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/C/ed3ec1d5ae5e4da5a297370347b5db2b to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/C/ed3ec1d5ae5e4da5a297370347b5db2b 2023-04-16 00:54:14,048 DEBUG [StoreCloser-TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/C/048858e9a71e4b60a0689027894b6451 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/C/048858e9a71e4b60a0689027894b6451 2023-04-16 00:54:14,049 DEBUG [StoreCloser-TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/C/e1a2b1285d9c41df841a226d149eaf57 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/C/e1a2b1285d9c41df841a226d149eaf57 2023-04-16 00:54:14,051 DEBUG [StoreCloser-TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/C/22ae222c4480457d84e9bef4e411489e to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/C/22ae222c4480457d84e9bef4e411489e 2023-04-16 00:54:14,052 DEBUG [StoreCloser-TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/C/4f0d2eafb2174fa495279f45b84c02d5 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/C/4f0d2eafb2174fa495279f45b84c02d5 2023-04-16 00:54:14,053 DEBUG [StoreCloser-TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/C/5d8b41609f4e42a18e4d23e8c688880c to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/C/5d8b41609f4e42a18e4d23e8c688880c 2023-04-16 00:54:14,055 DEBUG [StoreCloser-TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/C/68fe0c007b144c26af83d9c2ef895eec to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/C/68fe0c007b144c26af83d9c2ef895eec 2023-04-16 00:54:14,056 DEBUG [StoreCloser-TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/C/5cda4afcb1f8485ab1ef5c121276c90c to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/C/5cda4afcb1f8485ab1ef5c121276c90c 2023-04-16 00:54:14,057 DEBUG [StoreCloser-TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/C/79bfe09a6a5f49d5a49988b3191a255f to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/C/79bfe09a6a5f49d5a49988b3191a255f 2023-04-16 00:54:14,059 DEBUG [StoreCloser-TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/C/afb3b38579dc4f0fb532aa79bf8ca79c to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/C/afb3b38579dc4f0fb532aa79bf8ca79c 2023-04-16 00:54:14,060 DEBUG [StoreCloser-TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/C/c2040e6802c44108be891184e85171ed to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/C/c2040e6802c44108be891184e85171ed 2023-04-16 00:54:14,061 DEBUG [StoreCloser-TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/C/64e168accb044b7aa27fa8331d3872fb to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/C/64e168accb044b7aa27fa8331d3872fb 2023-04-16 00:54:14,062 DEBUG [StoreCloser-TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/C/c0940893fadc4e2b8a120177cc1e09cc to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/C/c0940893fadc4e2b8a120177cc1e09cc 2023-04-16 00:54:14,063 DEBUG [StoreCloser-TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/C/a4d87411c83443929fdbe9b819ec40a4 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/C/a4d87411c83443929fdbe9b819ec40a4 2023-04-16 00:54:14,064 DEBUG [StoreCloser-TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/C/648f5dd20c6749969b2473193cf7877d to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/C/648f5dd20c6749969b2473193cf7877d 2023-04-16 00:54:14,065 DEBUG [StoreCloser-TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/C/7834ad522c7041afaefd26df423307c6 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/C/7834ad522c7041afaefd26df423307c6 2023-04-16 00:54:14,066 DEBUG [StoreCloser-TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/C/4c108f2e62ec46eb8a0c6d025b65a751 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/C/4c108f2e62ec46eb8a0c6d025b65a751 2023-04-16 00:54:14,068 DEBUG [StoreCloser-TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/C/c9d1fa345c1844768b0413a43117c79b to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/C/c9d1fa345c1844768b0413a43117c79b 2023-04-16 00:54:14,072 DEBUG [StoreCloser-TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/C/ef328db1c5a34ce5af689ae0a510bedd to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/C/ef328db1c5a34ce5af689ae0a510bedd 2023-04-16 00:54:14,073 DEBUG [StoreCloser-TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/C/f8394a06624343caa2bf05ef89377797 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/C/f8394a06624343caa2bf05ef89377797 2023-04-16 00:54:14,075 DEBUG [StoreCloser-TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/C/3b547002069244838023503d420fefc6 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/C/3b547002069244838023503d420fefc6 2023-04-16 00:54:14,076 DEBUG [StoreCloser-TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/C/66010489386f41e886412436ace43af5 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/C/66010489386f41e886412436ace43af5 2023-04-16 00:54:14,077 DEBUG [StoreCloser-TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/C/ab605045f2fa4eab8ab381778c36558a to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/C/ab605045f2fa4eab8ab381778c36558a 2023-04-16 00:54:14,078 DEBUG [StoreCloser-TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/C/06fcfe628b674f3f81787dea2de9baea to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/C/06fcfe628b674f3f81787dea2de9baea 2023-04-16 00:54:14,083 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/recovered.edits/921.seqid, newMaxSeqId=921, maxSeqId=1 2023-04-16 00:54:14,085 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba. 2023-04-16 00:54:14,085 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for c9039369968c422a52a2c7db83b430ba: 2023-04-16 00:54:14,087 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed c9039369968c422a52a2c7db83b430ba 2023-04-16 00:54:14,087 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=13 updating hbase:meta row=c9039369968c422a52a2c7db83b430ba, regionState=CLOSED 2023-04-16 00:54:14,087 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681606454087"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681606454087"}]},"ts":"1681606454087"} 2023-04-16 00:54:14,092 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=14, resume processing ppid=13 2023-04-16 00:54:14,092 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=14, ppid=13, state=SUCCESS; CloseRegionProcedure c9039369968c422a52a2c7db83b430ba, server=jenkins-hbase4.apache.org,34975,1681606419217 in 819 msec 2023-04-16 00:54:14,094 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=13, resume processing ppid=12 2023-04-16 00:54:14,094 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=13, ppid=12, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=c9039369968c422a52a2c7db83b430ba, UNASSIGN in 827 msec 2023-04-16 00:54:14,095 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681606454095"}]},"ts":"1681606454095"} 2023-04-16 00:54:14,096 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-16 00:54:14,098 INFO [PEWorker-2] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-16 00:54:14,100 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=12, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 928 msec 2023-04-16 00:54:14,284 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-16 00:54:14,284 INFO [Listener at localhost/33477] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 12 completed 2023-04-16 00:54:14,289 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete TestAcidGuarantees 2023-04-16 00:54:14,294 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure2.ProcedureExecutor(1029): Stored pid=15, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-16 00:54:14,296 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=15, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-16 00:54:14,297 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=15, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-16 00:54:14,299 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1227): Checking to see if procedure is done pid=15 2023-04-16 00:54:14,302 DEBUG [HFileArchiver-3] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/.tmp/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba 2023-04-16 00:54:14,304 DEBUG [HFileArchiver-3] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/.tmp/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/A, FileablePath, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/.tmp/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/B, FileablePath, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/.tmp/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/C, FileablePath, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/.tmp/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/recovered.edits] 2023-04-16 00:54:14,307 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/.tmp/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/A/13b5e13d95664140a71bea03190020f9 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/A/13b5e13d95664140a71bea03190020f9 2023-04-16 00:54:14,309 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/.tmp/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/A/67b53b07717a49f892f7c176c0c45cbe to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/A/67b53b07717a49f892f7c176c0c45cbe 2023-04-16 00:54:14,310 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/.tmp/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/A/c1e38ed24eab4581adb7f71eb9f92d0f to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/A/c1e38ed24eab4581adb7f71eb9f92d0f 2023-04-16 00:54:14,313 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/.tmp/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/B/0d7728f518a34caeb0ee8eea52dd5f42 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/B/0d7728f518a34caeb0ee8eea52dd5f42 2023-04-16 00:54:14,314 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/.tmp/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/B/2065e8b8c5924567947d4c030cfcebc1 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/B/2065e8b8c5924567947d4c030cfcebc1 2023-04-16 00:54:14,315 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/.tmp/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/B/39cfcae02ea84e4abd283f061ea6cc96 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/B/39cfcae02ea84e4abd283f061ea6cc96 2023-04-16 00:54:14,318 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/.tmp/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/C/107040c091bf4ccf89371531d217a8c8 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/C/107040c091bf4ccf89371531d217a8c8 2023-04-16 00:54:14,319 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/.tmp/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/C/9f24b31541124567b0799ab0e3f49f11 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/C/9f24b31541124567b0799ab0e3f49f11 2023-04-16 00:54:14,320 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/.tmp/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/C/b52dc209eb3049bda69feb5f41803903 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/C/b52dc209eb3049bda69feb5f41803903 2023-04-16 00:54:14,323 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/.tmp/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/recovered.edits/921.seqid to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba/recovered.edits/921.seqid 2023-04-16 00:54:14,323 DEBUG [HFileArchiver-3] backup.HFileArchiver(596): Deleted hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/.tmp/data/default/TestAcidGuarantees/c9039369968c422a52a2c7db83b430ba 2023-04-16 00:54:14,323 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-16 00:54:14,325 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=15, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-16 00:54:14,345 WARN [PEWorker-5] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-16 00:54:14,386 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-16 00:54:14,387 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=15, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-16 00:54:14,387 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-16 00:54:14,388 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1681606454387"}]},"ts":"9223372036854775807"} 2023-04-16 00:54:14,390 INFO [PEWorker-5] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-16 00:54:14,390 DEBUG [PEWorker-5] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => c9039369968c422a52a2c7db83b430ba, NAME => 'TestAcidGuarantees,,1681606423664.c9039369968c422a52a2c7db83b430ba.', STARTKEY => '', ENDKEY => ''}] 2023-04-16 00:54:14,390 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-16 00:54:14,390 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1681606454390"}]},"ts":"9223372036854775807"} 2023-04-16 00:54:14,392 INFO [PEWorker-5] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-16 00:54:14,394 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(130): Finished pid=15, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-16 00:54:14,395 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=15, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 104 msec 2023-04-16 00:54:14,399 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1227): Checking to see if procedure is done pid=15 2023-04-16 00:54:14,400 INFO [Listener at localhost/33477] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 15 completed 2023-04-16 00:54:14,415 INFO [Listener at localhost/33477] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testMixedAtomicity Thread=230 (was 206) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-7 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x4a6b0192-shared-pool-12 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: (jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:458) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:924) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-3 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-9 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:458) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:924) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-13 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-3 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-10 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-12 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Timer for 'HBase' metrics system java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-5 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x4a6b0192-shared-pool-9 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-4 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-2 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:883) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS:0;jenkins-hbase4:34975-shortCompactions-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.PriorityBlockingQueue.take(PriorityBlockingQueue.java:549) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x4a6b0192-shared-pool-10 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x4a6b0192-shared-pool-11 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-8 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-11 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-1763287748_15 at /127.0.0.1:45252 [Waiting for operation #3] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RSProcedureDispatcher-pool-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-1-3 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-6 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) - Thread LEAK? -, OpenFileDescriptor=575 (was 470) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=334 (was 227) - SystemLoadAverage LEAK? -, ProcessCount=171 (was 171), AvailableMemoryMB=878 (was 1619) 2023-04-16 00:54:14,426 INFO [Listener at localhost/33477] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testMobMixedAtomicity Thread=230, OpenFileDescriptor=575, MaxFileDescriptor=60000, SystemLoadAverage=334, ProcessCount=171, AvailableMemoryMB=878 2023-04-16 00:54:14,428 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-16 00:54:14,428 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-16 00:54:14,429 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure2.ProcedureExecutor(1029): Stored pid=16, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-16 00:54:14,430 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-16 00:54:14,431 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 16 2023-04-16 00:54:14,431 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-16 00:54:14,431 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-04-16 00:54:14,433 DEBUG [HFileArchiver-4] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/.tmp/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f 2023-04-16 00:54:14,433 DEBUG [HFileArchiver-4] backup.HFileArchiver(153): Directory hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/.tmp/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f empty. 2023-04-16 00:54:14,434 DEBUG [HFileArchiver-4] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/.tmp/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f 2023-04-16 00:54:14,434 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-16 00:54:14,532 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-04-16 00:54:14,733 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-04-16 00:54:14,842 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-16 00:54:14,844 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => bcfff279d7c9f90f5c57aef887110e5f, NAME => 'TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/.tmp 2023-04-16 00:54:15,034 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-04-16 00:54:15,250 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-16 00:54:15,250 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing bcfff279d7c9f90f5c57aef887110e5f, disabling compactions & flushes 2023-04-16 00:54:15,250 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f. 2023-04-16 00:54:15,250 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f. 2023-04-16 00:54:15,250 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f. after waiting 0 ms 2023-04-16 00:54:15,250 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f. 2023-04-16 00:54:15,250 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f. 2023-04-16 00:54:15,250 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for bcfff279d7c9f90f5c57aef887110e5f: 2023-04-16 00:54:15,252 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-16 00:54:15,253 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681606455253"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681606455253"}]},"ts":"1681606455253"} 2023-04-16 00:54:15,255 INFO [PEWorker-1] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-16 00:54:15,256 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-16 00:54:15,256 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681606455256"}]},"ts":"1681606455256"} 2023-04-16 00:54:15,257 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-16 00:54:15,261 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=17, ppid=16, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=bcfff279d7c9f90f5c57aef887110e5f, ASSIGN}] 2023-04-16 00:54:15,262 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=17, ppid=16, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=bcfff279d7c9f90f5c57aef887110e5f, ASSIGN 2023-04-16 00:54:15,263 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=17, ppid=16, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=bcfff279d7c9f90f5c57aef887110e5f, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,34975,1681606419217; forceNewPlan=false, retain=false 2023-04-16 00:54:15,414 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=17 updating hbase:meta row=bcfff279d7c9f90f5c57aef887110e5f, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:15,414 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681606455414"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681606455414"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681606455414"}]},"ts":"1681606455414"} 2023-04-16 00:54:15,416 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=18, ppid=17, state=RUNNABLE; OpenRegionProcedure bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217}] 2023-04-16 00:54:15,535 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-04-16 00:54:15,571 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f. 2023-04-16 00:54:15,571 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => bcfff279d7c9f90f5c57aef887110e5f, NAME => 'TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f.', STARTKEY => '', ENDKEY => ''} 2023-04-16 00:54:15,571 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees bcfff279d7c9f90f5c57aef887110e5f 2023-04-16 00:54:15,572 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-16 00:54:15,572 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for bcfff279d7c9f90f5c57aef887110e5f 2023-04-16 00:54:15,572 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for bcfff279d7c9f90f5c57aef887110e5f 2023-04-16 00:54:15,573 INFO [StoreOpener-bcfff279d7c9f90f5c57aef887110e5f-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region bcfff279d7c9f90f5c57aef887110e5f 2023-04-16 00:54:15,574 DEBUG [StoreOpener-bcfff279d7c9f90f5c57aef887110e5f-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/A 2023-04-16 00:54:15,574 DEBUG [StoreOpener-bcfff279d7c9f90f5c57aef887110e5f-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/A 2023-04-16 00:54:15,575 INFO [StoreOpener-bcfff279d7c9f90f5c57aef887110e5f-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-16 00:54:15,575 INFO [StoreOpener-bcfff279d7c9f90f5c57aef887110e5f-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region bcfff279d7c9f90f5c57aef887110e5f columnFamilyName A 2023-04-16 00:54:15,576 INFO [StoreOpener-bcfff279d7c9f90f5c57aef887110e5f-1] regionserver.HStore(310): Store=bcfff279d7c9f90f5c57aef887110e5f/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-16 00:54:15,576 INFO [StoreOpener-bcfff279d7c9f90f5c57aef887110e5f-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region bcfff279d7c9f90f5c57aef887110e5f 2023-04-16 00:54:15,577 DEBUG [StoreOpener-bcfff279d7c9f90f5c57aef887110e5f-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/B 2023-04-16 00:54:15,577 DEBUG [StoreOpener-bcfff279d7c9f90f5c57aef887110e5f-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/B 2023-04-16 00:54:15,577 INFO [StoreOpener-bcfff279d7c9f90f5c57aef887110e5f-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-16 00:54:15,578 INFO [StoreOpener-bcfff279d7c9f90f5c57aef887110e5f-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region bcfff279d7c9f90f5c57aef887110e5f columnFamilyName B 2023-04-16 00:54:15,578 INFO [StoreOpener-bcfff279d7c9f90f5c57aef887110e5f-1] regionserver.HStore(310): Store=bcfff279d7c9f90f5c57aef887110e5f/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-16 00:54:15,578 INFO [StoreOpener-bcfff279d7c9f90f5c57aef887110e5f-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region bcfff279d7c9f90f5c57aef887110e5f 2023-04-16 00:54:15,580 DEBUG [StoreOpener-bcfff279d7c9f90f5c57aef887110e5f-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/C 2023-04-16 00:54:15,580 DEBUG [StoreOpener-bcfff279d7c9f90f5c57aef887110e5f-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/C 2023-04-16 00:54:15,580 INFO [StoreOpener-bcfff279d7c9f90f5c57aef887110e5f-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-16 00:54:15,580 INFO [StoreOpener-bcfff279d7c9f90f5c57aef887110e5f-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region bcfff279d7c9f90f5c57aef887110e5f columnFamilyName C 2023-04-16 00:54:15,581 INFO [StoreOpener-bcfff279d7c9f90f5c57aef887110e5f-1] regionserver.HStore(310): Store=bcfff279d7c9f90f5c57aef887110e5f/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-16 00:54:15,581 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f. 2023-04-16 00:54:15,582 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f 2023-04-16 00:54:15,582 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f 2023-04-16 00:54:15,584 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-16 00:54:15,585 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for bcfff279d7c9f90f5c57aef887110e5f 2023-04-16 00:54:15,587 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-16 00:54:15,587 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened bcfff279d7c9f90f5c57aef887110e5f; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=70141643, jitterRate=0.0451919287443161}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-16 00:54:15,587 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for bcfff279d7c9f90f5c57aef887110e5f: 2023-04-16 00:54:15,588 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f., pid=18, masterSystemTime=1681606455567 2023-04-16 00:54:15,589 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f. 2023-04-16 00:54:15,589 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f. 2023-04-16 00:54:15,590 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=17 updating hbase:meta row=bcfff279d7c9f90f5c57aef887110e5f, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:15,590 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681606455590"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1681606455590"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1681606455590"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1681606455590"}]},"ts":"1681606455590"} 2023-04-16 00:54:15,592 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=18, resume processing ppid=17 2023-04-16 00:54:15,593 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=18, ppid=17, state=SUCCESS; OpenRegionProcedure bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 in 175 msec 2023-04-16 00:54:15,594 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=17, resume processing ppid=16 2023-04-16 00:54:15,594 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=17, ppid=16, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=bcfff279d7c9f90f5c57aef887110e5f, ASSIGN in 332 msec 2023-04-16 00:54:15,594 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-16 00:54:15,595 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681606455595"}]},"ts":"1681606455595"} 2023-04-16 00:54:15,595 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-16 00:54:15,597 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-16 00:54:15,599 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=16, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 1.1690 sec 2023-04-16 00:54:16,536 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-04-16 00:54:16,536 INFO [Listener at localhost/33477] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 16 completed 2023-04-16 00:54:16,537 DEBUG [Listener at localhost/33477] zookeeper.ReadOnlyZKClient(139): Connect 0x33c943f0 to 127.0.0.1:53150 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-16 00:54:16,542 DEBUG [Listener at localhost/33477] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5b8d0eb2, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-16 00:54:16,543 DEBUG [Listener at localhost/33477] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-16 00:54:16,547 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:34662, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-16 00:54:16,548 DEBUG [Listener at localhost/33477] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-16 00:54:16,551 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:51770, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-16 00:54:16,558 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-16 00:54:16,558 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.HMaster$12(2454): Client=jenkins//172.31.14.131 modify table TestAcidGuarantees from 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} to 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', MOB_THRESHOLD => '4', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-16 00:54:16,566 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure2.ProcedureExecutor(1029): Stored pid=19, state=RUNNABLE:MODIFY_TABLE_PREPARE; ModifyTableProcedure table=TestAcidGuarantees 2023-04-16 00:54:16,979 DEBUG [PEWorker-4] util.FSTableDescriptors(570): Wrote into hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-16 00:54:16,980 DEBUG [PEWorker-4] util.FSTableDescriptors(519): Deleted hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-16 00:54:16,980 INFO [PEWorker-4] util.FSTableDescriptors(293): Updated tableinfo=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-16 00:54:16,984 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=20, ppid=19, state=RUNNABLE:REOPEN_TABLE_REGIONS_GET_REGIONS; ReopenTableRegionsProcedure table=TestAcidGuarantees}] 2023-04-16 00:54:16,996 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=21, ppid=20, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=bcfff279d7c9f90f5c57aef887110e5f, REOPEN/MOVE}] 2023-04-16 00:54:16,996 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=21, ppid=20, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=bcfff279d7c9f90f5c57aef887110e5f, REOPEN/MOVE 2023-04-16 00:54:16,997 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=bcfff279d7c9f90f5c57aef887110e5f, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:16,997 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681606456997"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681606456997"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681606456997"}]},"ts":"1681606456997"} 2023-04-16 00:54:16,998 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=22, ppid=21, state=RUNNABLE; CloseRegionProcedure bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217}] 2023-04-16 00:54:17,150 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close bcfff279d7c9f90f5c57aef887110e5f 2023-04-16 00:54:17,151 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing bcfff279d7c9f90f5c57aef887110e5f, disabling compactions & flushes 2023-04-16 00:54:17,151 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f. 2023-04-16 00:54:17,151 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f. 2023-04-16 00:54:17,151 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f. after waiting 0 ms 2023-04-16 00:54:17,151 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f. 2023-04-16 00:54:17,154 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-04-16 00:54:17,155 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f. 2023-04-16 00:54:17,155 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for bcfff279d7c9f90f5c57aef887110e5f: 2023-04-16 00:54:17,155 WARN [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(3509): Not adding moved region record: bcfff279d7c9f90f5c57aef887110e5f to self. 2023-04-16 00:54:17,157 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed bcfff279d7c9f90f5c57aef887110e5f 2023-04-16 00:54:17,157 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=bcfff279d7c9f90f5c57aef887110e5f, regionState=CLOSED 2023-04-16 00:54:17,157 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681606457157"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681606457157"}]},"ts":"1681606457157"} 2023-04-16 00:54:17,159 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=22, resume processing ppid=21 2023-04-16 00:54:17,159 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=22, ppid=21, state=SUCCESS; CloseRegionProcedure bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 in 160 msec 2023-04-16 00:54:17,160 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=21, ppid=20, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=bcfff279d7c9f90f5c57aef887110e5f, REOPEN/MOVE; state=CLOSED, location=jenkins-hbase4.apache.org,34975,1681606419217; forceNewPlan=false, retain=true 2023-04-16 00:54:17,310 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=bcfff279d7c9f90f5c57aef887110e5f, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:17,311 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681606457310"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681606457310"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681606457310"}]},"ts":"1681606457310"} 2023-04-16 00:54:17,312 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=23, ppid=21, state=RUNNABLE; OpenRegionProcedure bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217}] 2023-04-16 00:54:17,467 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f. 2023-04-16 00:54:17,467 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => bcfff279d7c9f90f5c57aef887110e5f, NAME => 'TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f.', STARTKEY => '', ENDKEY => ''} 2023-04-16 00:54:17,467 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees bcfff279d7c9f90f5c57aef887110e5f 2023-04-16 00:54:17,467 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-16 00:54:17,468 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for bcfff279d7c9f90f5c57aef887110e5f 2023-04-16 00:54:17,468 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for bcfff279d7c9f90f5c57aef887110e5f 2023-04-16 00:54:17,470 INFO [StoreOpener-bcfff279d7c9f90f5c57aef887110e5f-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region bcfff279d7c9f90f5c57aef887110e5f 2023-04-16 00:54:17,471 DEBUG [StoreOpener-bcfff279d7c9f90f5c57aef887110e5f-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/A 2023-04-16 00:54:17,471 DEBUG [StoreOpener-bcfff279d7c9f90f5c57aef887110e5f-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/A 2023-04-16 00:54:17,471 INFO [StoreOpener-bcfff279d7c9f90f5c57aef887110e5f-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-16 00:54:17,474 INFO [StoreOpener-bcfff279d7c9f90f5c57aef887110e5f-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region bcfff279d7c9f90f5c57aef887110e5f columnFamilyName A 2023-04-16 00:54:17,476 INFO [StoreOpener-bcfff279d7c9f90f5c57aef887110e5f-1] regionserver.HStore(310): Store=bcfff279d7c9f90f5c57aef887110e5f/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-16 00:54:17,476 INFO [StoreOpener-bcfff279d7c9f90f5c57aef887110e5f-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region bcfff279d7c9f90f5c57aef887110e5f 2023-04-16 00:54:17,477 DEBUG [StoreOpener-bcfff279d7c9f90f5c57aef887110e5f-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/B 2023-04-16 00:54:17,477 DEBUG [StoreOpener-bcfff279d7c9f90f5c57aef887110e5f-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/B 2023-04-16 00:54:17,477 INFO [StoreOpener-bcfff279d7c9f90f5c57aef887110e5f-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-16 00:54:17,478 INFO [StoreOpener-bcfff279d7c9f90f5c57aef887110e5f-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region bcfff279d7c9f90f5c57aef887110e5f columnFamilyName B 2023-04-16 00:54:17,478 INFO [StoreOpener-bcfff279d7c9f90f5c57aef887110e5f-1] regionserver.HStore(310): Store=bcfff279d7c9f90f5c57aef887110e5f/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-16 00:54:17,478 INFO [StoreOpener-bcfff279d7c9f90f5c57aef887110e5f-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region bcfff279d7c9f90f5c57aef887110e5f 2023-04-16 00:54:17,479 DEBUG [StoreOpener-bcfff279d7c9f90f5c57aef887110e5f-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/C 2023-04-16 00:54:17,479 DEBUG [StoreOpener-bcfff279d7c9f90f5c57aef887110e5f-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/C 2023-04-16 00:54:17,479 INFO [StoreOpener-bcfff279d7c9f90f5c57aef887110e5f-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-16 00:54:17,479 INFO [StoreOpener-bcfff279d7c9f90f5c57aef887110e5f-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region bcfff279d7c9f90f5c57aef887110e5f columnFamilyName C 2023-04-16 00:54:17,480 INFO [StoreOpener-bcfff279d7c9f90f5c57aef887110e5f-1] regionserver.HStore(310): Store=bcfff279d7c9f90f5c57aef887110e5f/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-16 00:54:17,480 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f. 2023-04-16 00:54:17,481 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f 2023-04-16 00:54:17,482 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f 2023-04-16 00:54:17,483 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-16 00:54:17,484 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for bcfff279d7c9f90f5c57aef887110e5f 2023-04-16 00:54:17,485 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened bcfff279d7c9f90f5c57aef887110e5f; next sequenceid=5; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=68964736, jitterRate=0.027654647827148438}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-16 00:54:17,485 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for bcfff279d7c9f90f5c57aef887110e5f: 2023-04-16 00:54:17,486 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f., pid=23, masterSystemTime=1681606457464 2023-04-16 00:54:17,487 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f. 2023-04-16 00:54:17,487 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f. 2023-04-16 00:54:17,488 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=bcfff279d7c9f90f5c57aef887110e5f, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:17,488 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681606457488"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1681606457488"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1681606457488"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1681606457488"}]},"ts":"1681606457488"} 2023-04-16 00:54:17,490 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=23, resume processing ppid=21 2023-04-16 00:54:17,490 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=23, ppid=21, state=SUCCESS; OpenRegionProcedure bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 in 177 msec 2023-04-16 00:54:17,492 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=21, resume processing ppid=20 2023-04-16 00:54:17,492 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=21, ppid=20, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=bcfff279d7c9f90f5c57aef887110e5f, REOPEN/MOVE in 495 msec 2023-04-16 00:54:17,494 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=20, resume processing ppid=19 2023-04-16 00:54:17,494 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=20, ppid=19, state=SUCCESS; ReopenTableRegionsProcedure table=TestAcidGuarantees in 510 msec 2023-04-16 00:54:17,496 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=19, state=SUCCESS; ModifyTableProcedure table=TestAcidGuarantees in 934 msec 2023-04-16 00:54:17,498 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1227): Checking to see if procedure is done pid=19 2023-04-16 00:54:17,504 DEBUG [Listener at localhost/33477] zookeeper.ReadOnlyZKClient(139): Connect 0x5b3934fb to 127.0.0.1:53150 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-16 00:54:17,510 DEBUG [Listener at localhost/33477] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@34135972, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-16 00:54:17,511 DEBUG [Listener at localhost/33477] zookeeper.ReadOnlyZKClient(139): Connect 0x1aa5eef5 to 127.0.0.1:53150 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-16 00:54:17,514 DEBUG [Listener at localhost/33477] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@a71edf0, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-16 00:54:17,514 DEBUG [Listener at localhost/33477] zookeeper.ReadOnlyZKClient(139): Connect 0x1441c9f0 to 127.0.0.1:53150 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-16 00:54:17,517 DEBUG [Listener at localhost/33477] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@21265fcb, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-16 00:54:17,518 DEBUG [Listener at localhost/33477] zookeeper.ReadOnlyZKClient(139): Connect 0x69f15cb4 to 127.0.0.1:53150 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-16 00:54:17,523 DEBUG [Listener at localhost/33477] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7aba8e46, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-16 00:54:17,523 DEBUG [Listener at localhost/33477] zookeeper.ReadOnlyZKClient(139): Connect 0x6c1c92c6 to 127.0.0.1:53150 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-16 00:54:17,527 DEBUG [Listener at localhost/33477] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@feb770d, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-16 00:54:17,527 DEBUG [Listener at localhost/33477] zookeeper.ReadOnlyZKClient(139): Connect 0x54451f11 to 127.0.0.1:53150 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-16 00:54:17,532 DEBUG [Listener at localhost/33477] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@c2598b, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-16 00:54:17,533 DEBUG [Listener at localhost/33477] zookeeper.ReadOnlyZKClient(139): Connect 0x2c4379c2 to 127.0.0.1:53150 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-16 00:54:17,536 DEBUG [Listener at localhost/33477] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@508246cc, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-16 00:54:17,536 DEBUG [Listener at localhost/33477] zookeeper.ReadOnlyZKClient(139): Connect 0x3f43b712 to 127.0.0.1:53150 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-16 00:54:17,539 DEBUG [Listener at localhost/33477] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@8795451, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-16 00:54:17,540 DEBUG [Listener at localhost/33477] zookeeper.ReadOnlyZKClient(139): Connect 0x7a3bb940 to 127.0.0.1:53150 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-16 00:54:17,549 DEBUG [Listener at localhost/33477] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@58e27a32, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-16 00:54:17,556 DEBUG [hconnection-0x7819e024-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-16 00:54:17,556 DEBUG [hconnection-0x587c4f6c-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-16 00:54:17,556 DEBUG [hconnection-0x674ff318-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-16 00:54:17,557 DEBUG [hconnection-0x509f0b2e-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-16 00:54:17,557 DEBUG [hconnection-0x189d7722-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-16 00:54:17,557 DEBUG [hconnection-0x2065fa01-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-16 00:54:17,557 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:34666, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-16 00:54:17,559 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:34680, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-16 00:54:17,559 DEBUG [hconnection-0x76506782-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-16 00:54:17,559 DEBUG [hconnection-0x49a54980-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-16 00:54:17,559 DEBUG [hconnection-0x6415d427-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-16 00:54:17,563 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:34702, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-16 00:54:17,563 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:34690, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-16 00:54:17,563 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:34718, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-16 00:54:17,563 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:34722, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-16 00:54:17,563 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:34714, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-16 00:54:17,564 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:54:17,565 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:34740, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-16 00:54:17,573 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:34736, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-16 00:54:17,580 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:54:17,581 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:54:17,581 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:54:17,581 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:54:17,581 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:54:17,581 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:54:17,581 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:17,581 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:17,583 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:17,583 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:54:17,583 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:54:17,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:54:17,585 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:17,585 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:54:17,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:17,585 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:17,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:54:17,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:17,586 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:54:17,592 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:54:17,592 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:54:17,592 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:54:17,593 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(9158): Flush requested on bcfff279d7c9f90f5c57aef887110e5f 2023-04-16 00:54:17,593 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:54:17,593 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:54:17,593 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing bcfff279d7c9f90f5c57aef887110e5f 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-16 00:54:17,596 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bcfff279d7c9f90f5c57aef887110e5f, store=A 2023-04-16 00:54:17,596 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:54:17,596 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bcfff279d7c9f90f5c57aef887110e5f, store=B 2023-04-16 00:54:17,596 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:54:17,596 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bcfff279d7c9f90f5c57aef887110e5f, store=C 2023-04-16 00:54:17,596 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:54:17,607 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:54:17,608 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:54:17,608 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f. 2023-04-16 00:54:17,608 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f. started... 2023-04-16 00:54:17,608 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f. as already flushing 2023-04-16 00:54:17,608 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f. 2023-04-16 00:54:17,608 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:54:17,608 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:54:17,608 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:17,610 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:54:17,610 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:54:17,611 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:17,611 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:17,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:17,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:17,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:17,612 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:17,612 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:54:17,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:17,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:17,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:17,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:17,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:17,614 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:54:17,614 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2ec20f17[Count = 0] remaining members to acquire global barrier 2023-04-16 00:54:17,614 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:54:17,614 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:17,615 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:17,616 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:17,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:17,616 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:54:17,616 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:54:17,616 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:54:17,616 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:17,616 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:54:17,618 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:54:17,618 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:54:17,618 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:17,618 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:54:17,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:17,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:17,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:17,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:17,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:17,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:17,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:17,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:17,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:17,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:17,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:54:17,621 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:54:17,621 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:54:17,621 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:54:17,621 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:54:17,621 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:54:17,621 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:54:17,622 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:17,622 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:17,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:17,622 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:17,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:17,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:17,622 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:54:17,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:17,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:17,623 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:54:17,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:54:17,623 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:17,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:17,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:17,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:17,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:17,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:17,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:17,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:17,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:17,627 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:17,629 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:54:17,630 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:54:17,630 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:17,630 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:54:17,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:54:17,630 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:17,630 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:17,630 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:17,630 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:17,630 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:17,630 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:17,630 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:54:17,630 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:54:17,630 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:54:17,630 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:54:17,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:54:17,630 DEBUG [Thread-984] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:54:17,630 DEBUG [Thread-984] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:54:17,661 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:17,661 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34740 deadline: 1681606517653, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:17,661 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:17,661 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34718 deadline: 1681606517654, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:17,662 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:17,662 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34680 deadline: 1681606517658, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:17,662 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:17,662 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:17,662 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34714 deadline: 1681606517656, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:17,662 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34702 deadline: 1681606517661, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:17,731 DEBUG [Thread-984] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:54:17,731 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:54:17,734 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:54:17,735 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:54:17,735 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:54:17,735 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:54:17,735 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:54:17,735 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:54:17,735 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:54:17,736 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:17,736 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:17,737 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:54:17,737 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:17,737 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:54:17,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:54:17,738 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:17,738 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:54:17,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:17,738 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:17,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:54:17,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:17,739 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:54:17,739 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:54:17,739 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:54:17,739 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:54:17,739 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:54:17,739 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:54:17,739 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:54:17,739 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:54:17,739 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f. 2023-04-16 00:54:17,739 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f. started... 2023-04-16 00:54:17,740 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f. as already flushing 2023-04-16 00:54:17,740 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f. 2023-04-16 00:54:17,740 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:54:17,740 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:54:17,740 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:17,740 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:54:17,740 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:54:17,742 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:17,742 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:17,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:17,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:17,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:17,743 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:17,743 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:54:17,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:17,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:17,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:17,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:17,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:17,744 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:54:17,744 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6eff7bf9[Count = 0] remaining members to acquire global barrier 2023-04-16 00:54:17,744 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:54:17,744 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:17,745 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:17,745 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:17,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:17,745 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:54:17,745 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:54:17,745 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:54:17,745 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:17,745 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:54:17,754 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:17,754 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:54:17,754 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:54:17,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:17,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:17,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:17,754 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:54:17,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:17,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:17,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:17,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:17,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:17,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:17,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:17,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:54:17,757 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:54:17,757 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:54:17,757 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:54:17,757 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:54:17,757 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:54:17,757 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:54:17,758 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:17,758 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:54:17,758 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:17,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:17,758 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:17,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:17,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:17,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:17,759 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:54:17,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:54:17,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:17,759 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:17,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:17,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:17,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:17,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:17,760 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:17,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:17,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:17,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:17,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:17,762 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:17,762 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:17,762 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:54:17,762 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:17,762 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:54:17,762 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:54:17,762 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:54:17,762 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:54:17,762 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:17,762 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:17,762 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:17,762 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:17,762 DEBUG [Thread-984] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:54:17,763 DEBUG [Thread-984] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:54:17,762 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:54:17,762 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:54:17,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:54:17,763 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:54:17,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:54:17,763 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:17,763 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34740 deadline: 1681606517763, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:17,765 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:17,765 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34714 deadline: 1681606517765, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:17,765 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:17,765 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:17,765 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34702 deadline: 1681606517765, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:17,766 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34718 deadline: 1681606517765, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:17,766 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:17,766 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34680 deadline: 1681606517765, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:17,863 DEBUG [Thread-984] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:54:17,863 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:54:17,865 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:54:17,884 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:54:17,884 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:54:17,884 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:54:17,884 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:54:17,884 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:54:17,884 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:54:17,885 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:17,885 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:17,887 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:17,887 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:54:17,887 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:54:17,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:54:17,887 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:17,888 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:54:17,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:17,888 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:17,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:54:17,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:17,888 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:54:17,896 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:54:17,896 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:54:17,896 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:54:17,896 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:54:17,896 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:54:17,900 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:54:17,900 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:54:17,900 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f. 2023-04-16 00:54:17,900 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f. started... 2023-04-16 00:54:17,900 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f. as already flushing 2023-04-16 00:54:17,900 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f. 2023-04-16 00:54:17,900 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:54:17,901 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:54:17,901 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:17,901 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:54:17,901 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:54:17,902 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:17,902 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:17,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:17,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:17,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:17,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:17,903 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:17,903 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:54:17,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:17,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:17,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:17,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:17,904 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:54:17,904 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@40fa5ef5[Count = 0] remaining members to acquire global barrier 2023-04-16 00:54:17,904 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:54:17,904 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:17,907 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:17,907 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:17,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:17,907 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:54:17,907 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:54:17,907 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:54:17,913 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:54:17,913 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:17,913 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:54:17,913 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:54:17,913 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:54:17,913 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:54:17,913 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:54:17,913 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:54:17,913 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:54:17,913 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:54:17,913 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:54:17,916 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:17,916 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:17,916 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:17,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:17,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:17,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:17,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:17,916 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:54:17,917 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:54:17,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:54:17,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:17,917 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:17,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:17,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:17,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:17,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:17,919 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:17,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:17,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:17,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:17,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:17,922 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:54:17,922 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:54:17,922 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:54:17,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:54:17,922 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:17,922 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:17,922 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:17,922 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:17,923 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:17,923 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:54:17,923 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:54:17,923 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:54:17,923 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:17,923 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:17,923 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:54:17,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:54:17,923 DEBUG [Thread-984] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:54:17,923 DEBUG [Thread-984] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:54:17,923 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:54:17,966 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:17,966 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34740 deadline: 1681606517965, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:17,968 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:17,968 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34714 deadline: 1681606517966, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:17,969 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:17,969 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34718 deadline: 1681606517967, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:17,970 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:17,970 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34702 deadline: 1681606517968, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:17,971 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:17,971 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34680 deadline: 1681606517969, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:18,023 DEBUG [Thread-984] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:54:18,024 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:54:18,026 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:54:18,028 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:54:18,028 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:54:18,028 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:54:18,028 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:54:18,028 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:54:18,028 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:54:18,029 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:18,029 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:18,030 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:54:18,030 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:18,030 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:54:18,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:54:18,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:18,030 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:18,030 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:54:18,030 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:18,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:54:18,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:18,031 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:54:18,032 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:54:18,032 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:54:18,032 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:54:18,032 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:54:18,032 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:54:18,036 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:54:18,036 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:54:18,036 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f. 2023-04-16 00:54:18,036 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f. started... 2023-04-16 00:54:18,036 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f. as already flushing 2023-04-16 00:54:18,036 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f. 2023-04-16 00:54:18,036 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:54:18,036 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:54:18,036 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:18,041 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:54:18,041 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:54:18,048 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:18,048 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:18,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:18,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:18,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:18,048 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230416e24073e8b2b044508bc4a46d85ba5881 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230416e24073e8b2b044508bc4a46d85ba5881 2023-04-16 00:54:18,048 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:18,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:18,048 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:54:18,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:18,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:18,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:18,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:18,051 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:54:18,051 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4ef80acf[Count = 0] remaining members to acquire global barrier 2023-04-16 00:54:18,051 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:54:18,051 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:18,054 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:18,054 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:18,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:18,054 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:54:18,054 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:54:18,054 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:54:18,054 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:18,054 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:54:18,055 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:54:18,055 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:54:18,055 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:18,055 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:54:18,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:18,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:18,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:18,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:18,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:18,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:18,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:18,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:18,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:18,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:18,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:54:18,060 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:54:18,060 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:54:18,060 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:54:18,060 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:54:18,060 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:54:18,060 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:54:18,062 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:18,062 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:54:18,062 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:18,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:18,062 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:18,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:18,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:18,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:18,063 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:54:18,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:54:18,064 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:18,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:18,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:18,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:18,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:18,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:18,069 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:18,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:18,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:18,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:18,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:18,071 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:54:18,071 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:54:18,072 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:54:18,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:54:18,072 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:54:18,071 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:18,072 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:54:18,072 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:18,072 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:18,072 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:18,072 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:54:18,072 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:18,072 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:18,072 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:18,072 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:54:18,072 DEBUG [Thread-984] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:54:18,072 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:54:18,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:54:18,073 DEBUG [Thread-984] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:54:18,173 DEBUG [Thread-984] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:54:18,174 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:54:18,176 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:54:18,188 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:54:18,188 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:54:18,188 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:54:18,188 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:54:18,188 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:54:18,188 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:54:18,189 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:18,189 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:18,191 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:54:18,191 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:18,191 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:54:18,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:54:18,191 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:18,191 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:54:18,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:18,192 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:18,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:54:18,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:18,192 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:54:18,196 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:54:18,196 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:54:18,196 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:54:18,196 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:54:18,196 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:54:18,200 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:54:18,200 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:54:18,200 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f. 2023-04-16 00:54:18,200 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f. started... 2023-04-16 00:54:18,200 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f. as already flushing 2023-04-16 00:54:18,200 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f. 2023-04-16 00:54:18,200 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:54:18,200 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:54:18,200 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:18,203 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:54:18,203 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:54:18,204 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:18,204 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:18,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:18,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:18,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:18,205 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:18,205 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:54:18,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:18,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:18,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:18,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:18,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:18,206 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:54:18,206 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@707c0eb0[Count = 0] remaining members to acquire global barrier 2023-04-16 00:54:18,206 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:54:18,206 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:18,208 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:18,208 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:18,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:18,208 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:54:18,208 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:54:18,208 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:54:18,209 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:18,209 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:54:18,210 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:18,210 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:54:18,210 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:54:18,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:18,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:18,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:18,210 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:54:18,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:18,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:18,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:18,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:18,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:18,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:18,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:18,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:54:18,213 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:54:18,213 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:54:18,214 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:54:18,214 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:54:18,214 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:54:18,214 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:54:18,220 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:18,220 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:18,220 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:54:18,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:18,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:18,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:18,220 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:18,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:18,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:18,221 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:54:18,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:54:18,221 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:18,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:18,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:18,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:18,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:18,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:18,222 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:18,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:18,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:18,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:18,225 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:18,225 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:18,225 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:18,225 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:18,225 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:18,225 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:18,225 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:18,225 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:54:18,225 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:54:18,225 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:54:18,225 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:54:18,225 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:54:18,225 DEBUG [Thread-984] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:54:18,225 DEBUG [Thread-984] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:54:18,225 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:54:18,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:54:18,226 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:54:18,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:54:18,269 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:18,269 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34740 deadline: 1681606518268, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:18,272 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:18,272 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34714 deadline: 1681606518272, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:18,273 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:18,273 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34718 deadline: 1681606518273, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:18,273 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:18,273 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34680 deadline: 1681606518273, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:18,274 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:18,274 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34702 deadline: 1681606518273, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:18,326 DEBUG [Thread-984] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:54:18,326 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:54:18,329 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:54:18,330 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:54:18,330 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:54:18,330 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:54:18,330 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:54:18,330 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:54:18,330 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:54:18,331 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:18,331 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:18,332 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:54:18,332 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:54:18,332 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:18,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:54:18,332 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:18,332 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:54:18,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:18,333 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:18,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:54:18,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:18,333 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:54:18,335 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:54:18,335 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:54:18,335 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:54:18,335 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:54:18,335 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:54:18,335 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:54:18,335 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:54:18,335 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f. 2023-04-16 00:54:18,335 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f. started... 2023-04-16 00:54:18,335 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f. as already flushing 2023-04-16 00:54:18,335 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f. 2023-04-16 00:54:18,335 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:54:18,335 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:54:18,335 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:18,335 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:54:18,335 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:54:18,338 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:18,338 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:18,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:18,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:18,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:18,339 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:18,339 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:54:18,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:18,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:18,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:18,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:18,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:18,341 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:54:18,341 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@65c993f5[Count = 0] remaining members to acquire global barrier 2023-04-16 00:54:18,341 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:54:18,341 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:18,342 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:18,342 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:18,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:18,342 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:54:18,342 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:54:18,342 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:18,342 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:54:18,342 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:54:18,343 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:54:18,343 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:54:18,343 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:54:18,344 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:18,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:18,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:18,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:18,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:18,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:18,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:18,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:18,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:18,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:18,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:18,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:54:18,346 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:54:18,346 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:54:18,346 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:54:18,346 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:54:18,346 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:54:18,346 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:54:18,348 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:18,348 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:54:18,348 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:18,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:18,348 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:18,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:18,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:18,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:18,348 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:54:18,348 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:18,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:54:18,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:18,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:18,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:18,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:18,351 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:18,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:18,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:18,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:18,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:18,357 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:18,357 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:54:18,357 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:18,358 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:54:18,358 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:54:18,358 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:54:18,358 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:54:18,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:54:18,358 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:54:18,358 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:18,358 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:18,358 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:18,358 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:18,358 DEBUG [Thread-984] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:54:18,358 DEBUG [Thread-984] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:54:18,358 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:54:18,358 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:54:18,358 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:18,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:54:18,458 DEBUG [Thread-984] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:54:18,459 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:54:18,461 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:54:18,462 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:54:18,463 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:54:18,463 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:54:18,463 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:54:18,463 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:54:18,463 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:54:18,464 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:18,464 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:18,465 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:54:18,465 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:18,465 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:54:18,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:54:18,472 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:18,472 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:54:18,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:18,473 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:18,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:54:18,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:18,473 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:54:18,474 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:54:18,474 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:54:18,474 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:54:18,474 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:54:18,474 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:54:18,474 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:54:18,474 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:54:18,474 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f. 2023-04-16 00:54:18,474 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f. started... 2023-04-16 00:54:18,474 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f. as already flushing 2023-04-16 00:54:18,474 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f. 2023-04-16 00:54:18,474 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:54:18,474 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:54:18,474 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:18,475 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:54:18,475 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:54:18,476 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:18,476 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:18,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:18,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:18,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:18,476 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:18,476 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:54:18,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:18,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:18,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:18,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:18,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:18,478 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:54:18,478 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:54:18,478 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7756744f[Count = 0] remaining members to acquire global barrier 2023-04-16 00:54:18,478 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:18,479 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:18,479 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:18,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:18,479 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:54:18,480 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:54:18,480 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:54:18,480 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:18,480 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:54:18,483 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=17, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/.tmp/A/e1d6985772b2459891a09ef7b6f7a71a 2023-04-16 00:54:18,486 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:18,487 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:54:18,487 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:54:18,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:18,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:18,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:18,487 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:54:18,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:18,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:18,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:18,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:18,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:18,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:18,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:18,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:54:18,494 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:54:18,494 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:54:18,494 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:54:18,494 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:54:18,494 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:54:18,494 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:54:18,496 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:18,496 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:54:18,496 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:18,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:18,496 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:18,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:18,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:18,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:18,497 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:18,497 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:54:18,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:54:18,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:18,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:18,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:18,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:18,499 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=17 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/.tmp/B/3f116ca5aea14ee089ffd2938d1a6e36 2023-04-16 00:54:18,499 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:18,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:18,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:18,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:18,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:18,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:18,502 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:18,503 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:18,503 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:18,503 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:18,503 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:18,503 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:18,503 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:18,503 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:54:18,503 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:54:18,503 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:54:18,503 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:54:18,503 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:54:18,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:54:18,503 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:54:18,503 DEBUG [Thread-984] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:54:18,503 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:54:18,503 DEBUG [Thread-984] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:54:18,504 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:54:18,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:54:18,604 DEBUG [Thread-984] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:54:18,604 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:54:18,610 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:54:18,613 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:54:18,613 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:54:18,613 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:54:18,613 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:54:18,613 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:54:18,613 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:54:18,613 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:18,613 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:18,615 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:54:18,615 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:18,615 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:54:18,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:54:18,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:18,615 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:18,615 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:54:18,616 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:18,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:54:18,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:18,616 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:54:18,618 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:54:18,618 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:54:18,618 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:54:18,618 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:54:18,618 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:54:18,618 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:54:18,618 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:54:18,618 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f. 2023-04-16 00:54:18,618 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f. started... 2023-04-16 00:54:18,618 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f. as already flushing 2023-04-16 00:54:18,618 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f. 2023-04-16 00:54:18,619 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:54:18,619 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:54:18,619 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:18,619 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:54:18,619 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:54:18,621 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:18,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:18,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:18,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:18,621 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:18,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:18,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:18,622 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:18,622 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:54:18,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:18,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:18,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:18,623 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:54:18,623 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:54:18,623 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7976a618[Count = 0] remaining members to acquire global barrier 2023-04-16 00:54:18,623 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:18,624 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:18,624 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:18,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:18,624 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:54:18,624 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:54:18,624 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:54:18,625 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:18,625 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:54:18,626 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:18,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:18,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:18,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:18,626 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:54:18,626 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:54:18,626 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:54:18,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:18,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:18,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:18,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:18,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:18,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:18,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:18,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:54:18,629 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:54:18,629 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:54:18,629 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:54:18,629 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:54:18,629 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:54:18,629 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:54:18,631 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:18,631 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:18,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:18,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:18,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:18,631 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:54:18,631 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:18,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:18,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:18,632 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:54:18,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:54:18,632 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:18,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:18,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:18,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:18,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:18,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:18,634 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:18,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:18,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:18,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:18,640 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:18,640 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:18,640 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:18,640 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:18,640 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:18,640 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:54:18,640 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:54:18,640 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:54:18,640 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:54:18,640 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:54:18,640 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:54:18,640 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:18,640 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:54:18,640 DEBUG [Thread-984] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:54:18,640 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:18,640 DEBUG [Thread-984] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:54:18,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:54:18,641 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:54:18,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:54:18,740 DEBUG [Thread-984] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:54:18,741 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:54:18,743 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:54:18,744 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:54:18,744 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:54:18,744 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:54:18,744 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:54:18,744 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:54:18,744 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:54:18,745 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:18,745 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:18,746 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:54:18,746 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:18,746 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:54:18,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:54:18,746 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:18,746 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:54:18,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:18,747 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:18,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:54:18,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:18,747 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:54:18,747 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:54:18,747 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:54:18,747 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:54:18,747 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:54:18,747 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:54:18,748 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:54:18,748 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:54:18,748 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f. 2023-04-16 00:54:18,748 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f. started... 2023-04-16 00:54:18,748 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f. as already flushing 2023-04-16 00:54:18,748 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f. 2023-04-16 00:54:18,748 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:54:18,748 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:54:18,748 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:18,748 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:54:18,748 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:54:18,749 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:18,749 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:18,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:18,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:18,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:18,750 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:18,750 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:54:18,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:18,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:18,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:18,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:18,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:18,751 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:54:18,751 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@69ef3e08[Count = 0] remaining members to acquire global barrier 2023-04-16 00:54:18,751 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:54:18,751 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:18,753 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:18,753 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:18,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:18,753 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:54:18,753 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:18,753 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:54:18,753 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:54:18,753 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:54:18,754 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:18,754 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:54:18,755 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:54:18,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:18,755 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:54:18,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:18,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:18,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:18,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:18,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:18,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:18,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:18,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:18,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:18,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:54:18,757 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:54:18,757 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:54:18,757 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:54:18,757 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:54:18,757 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:54:18,757 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:54:18,759 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:18,759 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:18,759 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:18,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:18,759 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:54:18,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:18,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:18,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:18,759 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:18,759 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:54:18,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:54:18,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:18,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:18,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:18,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:18,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:18,761 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:18,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:18,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:18,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:18,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:18,763 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:54:18,763 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:54:18,763 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:54:18,763 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:54:18,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:54:18,763 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:18,763 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:54:18,763 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:54:18,763 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:18,763 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:18,763 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:18,763 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:54:18,763 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:54:18,763 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:18,763 DEBUG [Thread-984] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:54:18,764 DEBUG [Thread-984] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:54:18,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:54:18,763 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:18,764 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:18,774 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:18,774 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34740 deadline: 1681606518773, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:18,776 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:18,776 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34702 deadline: 1681606518775, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:18,776 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:18,776 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34714 deadline: 1681606518775, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:18,777 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:18,777 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34680 deadline: 1681606518777, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:18,778 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:18,779 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34718 deadline: 1681606518778, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:18,864 DEBUG [Thread-984] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:54:18,864 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:54:18,866 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:54:18,867 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:54:18,867 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:54:18,867 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:54:18,867 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:54:18,867 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:54:18,867 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:54:18,868 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:18,868 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:18,870 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:54:18,870 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:18,870 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:54:18,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:54:18,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:18,871 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:18,871 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:54:18,871 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:18,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:54:18,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:18,872 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:54:18,872 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:54:18,872 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:54:18,872 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:54:18,872 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:54:18,872 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:54:18,873 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:54:18,873 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:54:18,873 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f. 2023-04-16 00:54:18,873 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f. started... 2023-04-16 00:54:18,873 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f. as already flushing 2023-04-16 00:54:18,873 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f. 2023-04-16 00:54:18,873 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:54:18,873 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:54:18,873 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:18,873 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:54:18,873 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:54:18,875 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:18,875 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:18,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:18,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:18,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:18,875 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:18,875 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:54:18,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:18,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:18,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:18,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:18,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:18,876 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:54:18,876 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@40105efb[Count = 0] remaining members to acquire global barrier 2023-04-16 00:54:18,876 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:54:18,877 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:18,877 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:18,878 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:18,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:18,878 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:54:18,878 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:54:18,878 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:54:18,878 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:18,878 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:54:18,879 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:18,879 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:54:18,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:18,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:18,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:18,879 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:54:18,879 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:54:18,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:18,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:18,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:18,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:18,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:18,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:18,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:18,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:54:18,882 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:54:18,882 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:54:18,882 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:54:18,882 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:54:18,882 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:54:18,882 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:54:18,884 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:18,884 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:18,884 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:54:18,884 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:18,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:18,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:18,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:18,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:18,885 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:54:18,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:54:18,885 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:18,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:18,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:18,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:18,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:18,886 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:18,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:18,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:18,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:18,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:18,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:18,888 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:18,888 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:18,888 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:18,888 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:54:18,888 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:18,888 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:54:18,888 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:18,889 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:54:18,889 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:18,889 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:54:18,889 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:54:18,889 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:54:18,889 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:18,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:54:18,889 DEBUG [Thread-984] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:54:18,889 DEBUG [Thread-984] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:54:18,889 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:54:18,889 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:54:18,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:54:18,918 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=17 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/.tmp/C/61b84698f9a14cf0869de0b18914632a 2023-04-16 00:54:18,923 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/.tmp/A/e1d6985772b2459891a09ef7b6f7a71a as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/A/e1d6985772b2459891a09ef7b6f7a71a 2023-04-16 00:54:18,928 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/A/e1d6985772b2459891a09ef7b6f7a71a, entries=150, sequenceid=17, filesize=25.1 K 2023-04-16 00:54:18,929 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/.tmp/B/3f116ca5aea14ee089ffd2938d1a6e36 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/B/3f116ca5aea14ee089ffd2938d1a6e36 2023-04-16 00:54:18,945 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/B/3f116ca5aea14ee089ffd2938d1a6e36, entries=150, sequenceid=17, filesize=11.6 K 2023-04-16 00:54:18,946 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/.tmp/C/61b84698f9a14cf0869de0b18914632a as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/C/61b84698f9a14cf0869de0b18914632a 2023-04-16 00:54:18,952 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/C/61b84698f9a14cf0869de0b18914632a, entries=150, sequenceid=17, filesize=11.6 K 2023-04-16 00:54:18,953 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for bcfff279d7c9f90f5c57aef887110e5f in 1360ms, sequenceid=17, compaction requested=false 2023-04-16 00:54:18,953 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for bcfff279d7c9f90f5c57aef887110e5f: 2023-04-16 00:54:18,989 DEBUG [Thread-984] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:54:18,990 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:54:18,993 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:54:18,995 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:54:18,995 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:54:18,995 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:54:18,995 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:54:18,995 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:54:18,995 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:54:18,996 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:18,996 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:18,998 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:54:18,998 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:18,998 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:54:18,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:54:18,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:18,999 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:18,999 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:54:18,999 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:19,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:54:19,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:19,000 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:54:19,000 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:54:19,000 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:54:19,000 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:54:19,000 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:54:19,001 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:54:19,008 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:54:19,008 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f. 2023-04-16 00:54:19,008 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f. started... 2023-04-16 00:54:19,008 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:54:19,008 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing bcfff279d7c9f90f5c57aef887110e5f 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-16 00:54:19,008 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bcfff279d7c9f90f5c57aef887110e5f, store=A 2023-04-16 00:54:19,009 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:54:19,009 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bcfff279d7c9f90f5c57aef887110e5f, store=B 2023-04-16 00:54:19,009 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:54:19,009 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bcfff279d7c9f90f5c57aef887110e5f, store=C 2023-04-16 00:54:19,009 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:54:19,050 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023041612f7b8e0a2be4939ad59eb743a1b227e to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041612f7b8e0a2be4939ad59eb743a1b227e 2023-04-16 00:54:19,058 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=36, memsize=35.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/.tmp/A/c3bc52b00b004f1ead6c43393355b4f9 2023-04-16 00:54:19,100 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=36 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/.tmp/B/004575861f934c2da4ab487a312b7e5d 2023-04-16 00:54:19,124 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=36 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/.tmp/C/f44186f6c28d4fce89bbc0768d661e1e 2023-04-16 00:54:19,131 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/.tmp/A/c3bc52b00b004f1ead6c43393355b4f9 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/A/c3bc52b00b004f1ead6c43393355b4f9 2023-04-16 00:54:19,140 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/A/c3bc52b00b004f1ead6c43393355b4f9, entries=150, sequenceid=36, filesize=25.1 K 2023-04-16 00:54:19,144 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/.tmp/B/004575861f934c2da4ab487a312b7e5d as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/B/004575861f934c2da4ab487a312b7e5d 2023-04-16 00:54:19,150 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/B/004575861f934c2da4ab487a312b7e5d, entries=150, sequenceid=36, filesize=11.6 K 2023-04-16 00:54:19,151 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/.tmp/C/f44186f6c28d4fce89bbc0768d661e1e as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/C/f44186f6c28d4fce89bbc0768d661e1e 2023-04-16 00:54:19,169 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/C/f44186f6c28d4fce89bbc0768d661e1e, entries=150, sequenceid=36, filesize=11.6 K 2023-04-16 00:54:19,170 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=0 B/0 for bcfff279d7c9f90f5c57aef887110e5f in 162ms, sequenceid=36, compaction requested=false 2023-04-16 00:54:19,170 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for bcfff279d7c9f90f5c57aef887110e5f: 2023-04-16 00:54:19,171 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f. 2023-04-16 00:54:19,171 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:54:19,171 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:54:19,171 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:19,171 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:54:19,171 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:54:19,173 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:19,173 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:19,173 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:19,173 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:54:19,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:19,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:19,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:19,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:19,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:19,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:19,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:19,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:19,176 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:54:19,176 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@420beba[Count = 0] remaining members to acquire global barrier 2023-04-16 00:54:19,176 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:54:19,176 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:19,178 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:19,178 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:19,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:19,178 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:54:19,178 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:54:19,178 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:54:19,178 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:19,178 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:54:19,180 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:19,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:19,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:19,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:19,180 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:54:19,180 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:54:19,180 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:54:19,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:19,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:19,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:19,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:19,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:19,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:19,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:19,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:54:19,182 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:54:19,182 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:54:19,183 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:54:19,183 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:54:19,183 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:54:19,183 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:54:19,184 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:19,184 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:54:19,184 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:19,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:19,184 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:19,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:19,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:19,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:19,185 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:54:19,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:54:19,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:19,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:19,185 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:19,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:19,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:19,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:19,187 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:19,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:19,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:19,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:19,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:19,194 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:54:19,194 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:19,194 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:54:19,194 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:54:19,194 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:19,194 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:54:19,194 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:54:19,194 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:19,194 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:19,195 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:19,195 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:19,195 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:19,194 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:54:19,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:54:19,195 DEBUG [Thread-984] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:54:19,195 DEBUG [Thread-984] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:54:19,195 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:54:19,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:54:19,295 DEBUG [Thread-984] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:54:19,296 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:54:19,298 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:54:19,300 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:54:19,300 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:54:19,300 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:54:19,300 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:54:19,300 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:54:19,300 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:54:19,300 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:19,300 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:19,301 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:54:19,301 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:19,302 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:54:19,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:54:19,302 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:19,302 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:54:19,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:19,306 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:19,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:54:19,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:19,307 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:54:19,307 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:54:19,307 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:54:19,307 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:54:19,307 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:54:19,307 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:54:19,308 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:54:19,308 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f. 2023-04-16 00:54:19,308 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f. started... 2023-04-16 00:54:19,308 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for bcfff279d7c9f90f5c57aef887110e5f: 2023-04-16 00:54:19,308 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f. 2023-04-16 00:54:19,308 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:54:19,309 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:54:19,309 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:19,309 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:54:19,309 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:54:19,310 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:19,310 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:19,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:19,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:19,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:19,311 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:19,311 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:54:19,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:19,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:19,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:19,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:19,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:19,312 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:54:19,313 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@482a9e40[Count = 0] remaining members to acquire global barrier 2023-04-16 00:54:19,313 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:54:19,313 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:19,314 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:19,314 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:19,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:19,314 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:54:19,314 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:54:19,314 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:54:19,314 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:19,314 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:54:19,317 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:54:19,317 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:54:19,317 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:54:19,317 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:19,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:19,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:19,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:19,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:19,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:19,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:19,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:19,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:19,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:19,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:19,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:54:19,320 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:54:19,320 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:54:19,320 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:54:19,320 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:54:19,320 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:54:19,321 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:54:19,322 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:19,322 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:54:19,322 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:19,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:19,322 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:19,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:19,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:19,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:19,322 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:54:19,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:54:19,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:19,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:19,323 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:19,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:19,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:19,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:19,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:19,325 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:19,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:19,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:19,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:19,327 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:19,327 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:19,327 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:54:19,327 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:54:19,327 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:54:19,327 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:54:19,327 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:54:19,327 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:19,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:54:19,327 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:54:19,327 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:54:19,327 DEBUG [Thread-984] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:54:19,327 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:19,328 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:54:19,328 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:19,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:54:19,328 DEBUG [Thread-984] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:54:19,328 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:19,328 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:19,428 DEBUG [Thread-984] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:54:19,429 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:54:19,431 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:54:19,432 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:54:19,432 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:54:19,433 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:54:19,433 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:54:19,433 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:54:19,433 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:54:19,433 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:19,433 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:19,434 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:19,434 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:54:19,435 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:54:19,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:54:19,435 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:19,435 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:54:19,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:19,435 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:19,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:54:19,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:19,436 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:54:19,436 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:54:19,436 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:54:19,436 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:54:19,436 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:54:19,436 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:54:19,437 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:54:19,437 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:54:19,437 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f. 2023-04-16 00:54:19,437 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f. started... 2023-04-16 00:54:19,437 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for bcfff279d7c9f90f5c57aef887110e5f: 2023-04-16 00:54:19,437 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f. 2023-04-16 00:54:19,438 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:54:19,438 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:54:19,438 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:19,438 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:54:19,438 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:54:19,440 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:19,440 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:19,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:19,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:19,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:19,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:19,441 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:19,441 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:54:19,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:19,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:19,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:19,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:19,442 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:54:19,443 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5825f633[Count = 0] remaining members to acquire global barrier 2023-04-16 00:54:19,443 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:54:19,443 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:19,444 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:19,444 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:19,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:19,444 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:54:19,444 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:54:19,444 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:54:19,444 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:19,444 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:54:19,445 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:54:19,445 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:54:19,445 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:19,445 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:54:19,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:19,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:19,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:19,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:19,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:19,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:19,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:19,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:19,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:19,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:19,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:54:19,449 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:54:19,449 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:54:19,449 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:54:19,449 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:54:19,449 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:54:19,449 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:54:19,450 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:19,450 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:19,451 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:19,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:19,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:19,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:19,450 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:54:19,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:19,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:19,451 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:54:19,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:54:19,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:19,451 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:19,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:19,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:19,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:19,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:19,453 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:19,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:19,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:19,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:19,457 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:19,457 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:54:19,457 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:19,457 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:54:19,457 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:54:19,457 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:54:19,457 DEBUG [Thread-984] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:54:19,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:54:19,457 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:54:19,457 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:19,458 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:19,458 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:19,458 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:19,458 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:19,457 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:54:19,458 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:54:19,457 DEBUG [Thread-984] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:54:19,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:54:19,558 DEBUG [Thread-984] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:54:19,559 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:54:19,561 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:54:19,562 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:54:19,562 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:54:19,562 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:54:19,562 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:54:19,562 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:54:19,562 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:54:19,563 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:19,563 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:19,564 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:19,564 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:54:19,564 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:54:19,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:54:19,564 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:19,564 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:54:19,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:19,565 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:19,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:54:19,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:19,565 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:54:19,566 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:54:19,566 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:54:19,566 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:54:19,566 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:54:19,566 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:54:19,566 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:54:19,566 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:54:19,567 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f. 2023-04-16 00:54:19,567 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f. started... 2023-04-16 00:54:19,568 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for bcfff279d7c9f90f5c57aef887110e5f: 2023-04-16 00:54:19,568 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f. 2023-04-16 00:54:19,568 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:54:19,568 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:54:19,568 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:19,568 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:54:19,568 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:54:19,570 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:19,570 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:19,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:19,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:19,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:19,571 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:19,571 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:54:19,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:19,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:19,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:19,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:19,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:19,574 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:54:19,574 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1a809c4a[Count = 0] remaining members to acquire global barrier 2023-04-16 00:54:19,574 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:54:19,574 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:19,575 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:19,575 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:19,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:19,575 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:54:19,575 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:54:19,575 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:54:19,575 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:19,575 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:54:19,578 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:19,578 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:54:19,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:19,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:19,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:19,578 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:54:19,578 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:54:19,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:19,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:19,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:19,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:19,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:19,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:19,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:19,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:54:19,581 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:54:19,581 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:54:19,581 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:54:19,581 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:54:19,581 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:54:19,581 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:54:19,583 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:19,583 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:19,583 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:54:19,583 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:19,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:19,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:19,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:19,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:19,584 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:19,584 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:54:19,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:54:19,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:19,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:19,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:19,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:19,585 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:19,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:19,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:19,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:19,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:19,588 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:19,588 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:19,588 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:19,588 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:54:19,588 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:19,588 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:54:19,588 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:54:19,588 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:54:19,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:54:19,588 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:19,588 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:19,588 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:19,588 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:54:19,588 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:54:19,588 DEBUG [Thread-984] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:54:19,588 DEBUG [Thread-984] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:54:19,589 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:54:19,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:54:19,589 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:54:19,642 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-16 00:54:19,689 DEBUG [Thread-984] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:54:19,689 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:54:19,692 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:54:19,696 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:54:19,696 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:54:19,696 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:54:19,696 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:54:19,696 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:54:19,696 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:54:19,697 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:19,697 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:19,698 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:54:19,698 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:19,698 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:54:19,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:54:19,698 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:19,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:19,698 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:54:19,699 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:19,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:54:19,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:19,699 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:54:19,699 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:54:19,699 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:54:19,700 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:54:19,700 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:54:19,700 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:54:19,704 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:54:19,704 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:54:19,704 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f. 2023-04-16 00:54:19,704 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f. started... 2023-04-16 00:54:19,704 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for bcfff279d7c9f90f5c57aef887110e5f: 2023-04-16 00:54:19,704 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f. 2023-04-16 00:54:19,704 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:54:19,704 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:54:19,704 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:19,704 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:54:19,704 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:54:19,712 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:19,712 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:19,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:19,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:19,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:19,712 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:19,712 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:54:19,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:19,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:19,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:19,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:19,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:19,714 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:54:19,714 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1b5424ce[Count = 0] remaining members to acquire global barrier 2023-04-16 00:54:19,714 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:54:19,714 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:19,716 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:19,716 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:19,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:19,717 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:54:19,717 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:54:19,717 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:19,717 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:54:19,717 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:54:19,718 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:19,718 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:54:19,718 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:54:19,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:19,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:19,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:19,718 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:54:19,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:19,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:19,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:19,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:19,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:19,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:19,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:19,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:54:19,721 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:54:19,721 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:54:19,722 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:54:19,722 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:54:19,722 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:54:19,722 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:54:19,724 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:19,724 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:54:19,724 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:19,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:19,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:19,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:19,724 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:19,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:19,724 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:54:19,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:54:19,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:19,724 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:19,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:19,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:19,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:19,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:19,726 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:19,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:19,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:19,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:19,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:19,728 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:19,728 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:19,728 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:19,728 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:19,728 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:54:19,728 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:54:19,728 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:54:19,729 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:54:19,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:54:19,729 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:54:19,728 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:54:19,728 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:19,729 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:54:19,729 DEBUG [Thread-984] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:54:19,729 DEBUG [Thread-984] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:54:19,729 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:54:19,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:54:19,729 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:19,729 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:19,824 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(9158): Flush requested on bcfff279d7c9f90f5c57aef887110e5f 2023-04-16 00:54:19,824 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing bcfff279d7c9f90f5c57aef887110e5f 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-16 00:54:19,825 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bcfff279d7c9f90f5c57aef887110e5f, store=A 2023-04-16 00:54:19,825 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:54:19,825 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bcfff279d7c9f90f5c57aef887110e5f, store=B 2023-04-16 00:54:19,825 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:54:19,825 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bcfff279d7c9f90f5c57aef887110e5f, store=C 2023-04-16 00:54:19,825 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:54:19,829 DEBUG [Thread-984] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:54:19,830 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:54:19,835 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:54:19,844 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:54:19,844 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:54:19,844 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:54:19,844 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:54:19,844 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:54:19,844 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:54:19,845 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:19,845 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:19,847 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:54:19,847 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:19,848 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:54:19,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:54:19,848 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:19,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:19,848 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:54:19,848 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:19,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:54:19,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:19,849 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:54:19,852 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:54:19,852 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:54:19,852 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:54:19,852 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:54:19,852 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:54:19,853 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:54:19,853 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:54:19,853 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f. 2023-04-16 00:54:19,853 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f. started... 2023-04-16 00:54:19,853 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f. as already flushing 2023-04-16 00:54:19,853 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f. 2023-04-16 00:54:19,853 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:54:19,853 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:54:19,853 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:19,853 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:54:19,853 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:54:19,855 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:19,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:19,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:19,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:19,855 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:19,859 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:19,859 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:54:19,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:19,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:19,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:19,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:19,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:19,861 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:54:19,861 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@790617a[Count = 0] remaining members to acquire global barrier 2023-04-16 00:54:19,861 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:54:19,862 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:19,863 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:19,863 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:19,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:19,863 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:54:19,863 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:54:19,863 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:54:19,864 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:19,864 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:54:19,865 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:54:19,865 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:54:19,865 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:19,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:19,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:19,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:19,865 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:54:19,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:19,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:19,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:19,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:19,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:19,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:19,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:19,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:54:19,869 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:54:19,869 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:54:19,869 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:54:19,869 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:54:19,869 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:54:19,869 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:54:19,871 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:19,871 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:19,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:19,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:19,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:19,872 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:19,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:19,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:19,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:19,872 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:54:19,873 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:54:19,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:54:19,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:19,874 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=359, cellSize=16435, totalHeapSize=37576, min timestamp=1681606459859, max timestamp=1681606459872Num uniques 859; 2023-04-16 00:54:19,874 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=16435, suffix heap size=117064,new segment heap size=37576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=359 2023-04-16 00:54:19,874 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:19,875 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=359, cellSize=16435, totalHeapSize=37576, min timestamp=1681606459859, max timestamp=1681606459872Num uniques 859; 2023-04-16 00:54:19,875 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=16435, suffix heap size=117064,new segment heap size=37576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=359 2023-04-16 00:54:19,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:19,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:19,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:19,876 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:19,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:19,876 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1681606459859, max timestamp=1681606459872Num uniques 859; 2023-04-16 00:54:19,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:19,876 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-16 00:54:19,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:19,880 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:54:19,880 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:54:19,880 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:54:19,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:54:19,880 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:19,880 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:19,880 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:19,880 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:54:19,880 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:54:19,881 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:54:19,880 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:19,881 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:19,881 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:19,881 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:54:19,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:54:19,881 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:19,881 DEBUG [Thread-984] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:54:19,881 DEBUG [Thread-984] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:54:19,881 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:54:19,899 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:19,899 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34718 deadline: 1681606519891, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:19,899 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:19,899 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34740 deadline: 1681606519892, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:19,899 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:19,899 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34714 deadline: 1681606519893, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:19,900 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:19,900 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34680 deadline: 1681606519893, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:19,902 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:19,902 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34702 deadline: 1681606519899, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:19,981 DEBUG [Thread-984] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:54:19,982 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:54:19,984 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:54:19,985 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:54:19,985 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:54:19,985 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:54:19,985 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:54:19,985 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:54:19,985 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:54:19,985 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:19,985 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:19,987 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:54:19,987 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:19,987 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:54:19,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:54:19,987 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:19,987 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:54:19,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:19,988 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:19,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:54:19,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:19,988 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:54:19,988 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:54:19,988 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:54:19,988 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:54:19,988 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:54:19,989 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:54:19,989 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:54:19,989 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:54:19,989 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f. 2023-04-16 00:54:19,989 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f. started... 2023-04-16 00:54:19,989 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f. as already flushing 2023-04-16 00:54:19,989 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f. 2023-04-16 00:54:19,989 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:54:19,989 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:54:19,989 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:19,989 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:54:19,989 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:54:19,991 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:19,991 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:19,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:19,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:19,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:19,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:19,991 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:19,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:19,991 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:54:19,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:19,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:19,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:19,993 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:54:19,993 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@70739bd5[Count = 0] remaining members to acquire global barrier 2023-04-16 00:54:19,993 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:54:19,993 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:19,995 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:19,995 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:19,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:19,995 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:54:19,996 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:54:19,996 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:19,996 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:54:19,996 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:54:19,997 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:54:19,997 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:54:19,997 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:19,997 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:54:19,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:19,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:19,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:19,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:19,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:19,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:19,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:19,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:19,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:19,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:19,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:54:19,999 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:54:19,999 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:54:19,999 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:54:19,999 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:54:19,999 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:54:19,999 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:54:20,001 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:20,001 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:20,001 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:20,001 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34718 deadline: 1681606520000, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:20,001 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:54:20,001 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:20,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:20,001 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:20,002 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:20,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:20,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:20,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:20,002 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34714 deadline: 1681606520000, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:20,002 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:20,002 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34740 deadline: 1681606520000, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:20,002 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:20,002 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34680 deadline: 1681606520001, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:20,002 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:54:20,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:20,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:54:20,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:20,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:20,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:20,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:20,003 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:20,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:20,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:20,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:20,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:20,006 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:20,006 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:54:20,006 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:54:20,006 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:54:20,006 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:54:20,006 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:54:20,006 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:54:20,006 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:20,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:54:20,006 DEBUG [Thread-984] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:54:20,006 DEBUG [Thread-984] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:54:20,006 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:20,006 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:54:20,006 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:54:20,006 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:20,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:54:20,006 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:20,007 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:20,007 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:20,011 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:20,011 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34702 deadline: 1681606520004, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:20,106 DEBUG [Thread-984] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:54:20,107 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:54:20,109 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:54:20,110 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:54:20,110 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:54:20,110 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:54:20,110 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:54:20,110 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:54:20,110 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:54:20,111 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:20,111 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:20,113 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:54:20,113 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:20,113 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:54:20,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:54:20,113 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:20,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:20,113 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:54:20,114 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:20,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:54:20,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:20,114 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:54:20,114 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:54:20,114 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:54:20,114 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:54:20,114 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:54:20,115 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:54:20,115 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:54:20,115 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:54:20,115 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f. 2023-04-16 00:54:20,115 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f. started... 2023-04-16 00:54:20,115 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f. as already flushing 2023-04-16 00:54:20,115 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f. 2023-04-16 00:54:20,115 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:54:20,115 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:54:20,115 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:20,115 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:54:20,115 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:54:20,116 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:20,116 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:20,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:20,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:20,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:20,117 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:20,117 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:54:20,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:20,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:20,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:20,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:20,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:20,118 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:54:20,118 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:54:20,118 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6e0f3b5d[Count = 0] remaining members to acquire global barrier 2023-04-16 00:54:20,118 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:20,122 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:20,122 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:20,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:20,122 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:54:20,122 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:54:20,122 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:54:20,122 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:20,122 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:54:20,125 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:54:20,125 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:20,126 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:54:20,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:20,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:20,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:20,126 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:54:20,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:20,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:20,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:20,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:20,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:20,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:20,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:20,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:54:20,128 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:54:20,128 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:54:20,128 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:54:20,128 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:54:20,128 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:54:20,128 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:54:20,130 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:20,130 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:20,130 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:20,130 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:54:20,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:20,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:20,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:20,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:20,130 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:54:20,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:54:20,131 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:20,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:20,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:20,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:20,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:20,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:20,132 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:20,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:20,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:20,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:20,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:20,134 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:20,134 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:20,134 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:20,134 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:54:20,135 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:54:20,134 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:20,135 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:54:20,135 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:54:20,135 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:54:20,135 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:54:20,135 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:20,135 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:20,135 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:20,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:54:20,135 DEBUG [Thread-984] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:54:20,135 DEBUG [Thread-984] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:54:20,135 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:54:20,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:54:20,204 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:20,205 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34740 deadline: 1681606520203, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:20,205 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:20,205 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34718 deadline: 1681606520204, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:20,205 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:20,205 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34714 deadline: 1681606520204, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:20,205 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:20,206 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34680 deadline: 1681606520205, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:20,214 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:20,214 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34702 deadline: 1681606520212, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:20,235 DEBUG [Thread-984] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:54:20,236 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:54:20,237 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:54:20,239 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:54:20,239 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:54:20,239 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:54:20,239 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:54:20,239 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:54:20,239 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:54:20,239 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:20,239 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:20,241 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:54:20,241 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:54:20,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:54:20,241 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:20,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:20,241 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:20,241 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:54:20,242 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:20,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:54:20,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:20,242 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:54:20,242 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:54:20,242 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:54:20,242 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:54:20,242 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:54:20,242 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:54:20,243 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:54:20,243 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:54:20,243 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f. 2023-04-16 00:54:20,243 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f. started... 2023-04-16 00:54:20,243 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f. as already flushing 2023-04-16 00:54:20,243 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f. 2023-04-16 00:54:20,243 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:54:20,243 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:54:20,243 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:20,243 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:54:20,243 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:54:20,245 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:20,245 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:20,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:20,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:20,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:20,246 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:20,246 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:54:20,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:20,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:20,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:20,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:20,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:20,247 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:54:20,247 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:54:20,247 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@54c71006[Count = 0] remaining members to acquire global barrier 2023-04-16 00:54:20,247 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:20,248 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:20,248 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:20,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:20,249 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:54:20,249 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:54:20,249 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:20,249 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:54:20,249 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:54:20,250 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:54:20,250 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:54:20,250 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:20,250 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:54:20,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:20,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:20,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:20,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:20,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:20,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:20,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:20,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:20,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:20,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:20,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:54:20,253 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:54:20,253 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:54:20,253 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:54:20,253 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:54:20,253 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:54:20,253 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:54:20,255 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:20,255 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:20,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:20,255 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:20,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:20,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:20,255 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:54:20,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:20,255 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:20,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:20,255 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:54:20,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:54:20,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:20,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:20,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:20,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:20,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:20,257 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:20,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:20,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:20,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:20,261 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230416e2582097f7b44031b681effb499a685c to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230416e2582097f7b44031b681effb499a685c 2023-04-16 00:54:20,263 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:54:20,263 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:54:20,263 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:20,264 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:20,264 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:20,264 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:20,264 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:20,264 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:20,264 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:20,264 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:54:20,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:54:20,264 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:54:20,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:54:20,264 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:54:20,264 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:54:20,265 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:54:20,265 DEBUG [Thread-984] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:54:20,265 DEBUG [Thread-984] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:54:20,365 DEBUG [Thread-984] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:54:20,366 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:54:20,367 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:54:20,368 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:54:20,368 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:54:20,369 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:54:20,369 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:54:20,369 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:54:20,369 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:54:20,369 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:20,369 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:20,370 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:20,370 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:54:20,370 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:54:20,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:54:20,371 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:20,371 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:54:20,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:20,371 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:20,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:54:20,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:20,371 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:54:20,372 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:54:20,372 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:54:20,372 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:54:20,372 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:54:20,372 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:54:20,372 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:54:20,372 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:54:20,372 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f. 2023-04-16 00:54:20,372 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f. started... 2023-04-16 00:54:20,372 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f. as already flushing 2023-04-16 00:54:20,372 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f. 2023-04-16 00:54:20,372 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:54:20,372 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:54:20,372 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:20,372 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:54:20,372 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:54:20,374 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:20,374 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:20,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:20,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:20,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:20,374 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:20,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:20,374 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:54:20,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:20,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:20,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:20,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:20,375 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:54:20,375 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@50d82055[Count = 0] remaining members to acquire global barrier 2023-04-16 00:54:20,375 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:54:20,375 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:20,376 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:20,376 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:20,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:20,377 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:54:20,377 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:54:20,377 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:20,377 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:54:20,377 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:54:20,379 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:20,379 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:54:20,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:20,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:20,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:20,379 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:54:20,380 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:54:20,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:20,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:20,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:20,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:20,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:20,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:20,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:20,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:54:20,383 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:54:20,383 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:54:20,383 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:54:20,383 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:54:20,383 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:54:20,383 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:54:20,384 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:20,384 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:20,384 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:20,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:20,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:20,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:20,384 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:54:20,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:20,385 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:20,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:20,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:20,385 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:54:20,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:54:20,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:20,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:20,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:20,386 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:20,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:20,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:20,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:20,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:20,388 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:20,388 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:54:20,388 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:20,389 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:54:20,389 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:20,389 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:54:20,389 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:20,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:54:20,389 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:20,389 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:20,389 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:20,389 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:54:20,389 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:54:20,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:54:20,390 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:54:20,390 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:54:20,390 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:54:20,390 DEBUG [Thread-984] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:54:20,390 DEBUG [Thread-984] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:54:20,490 DEBUG [Thread-984] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:54:20,491 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:54:20,493 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:54:20,494 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:54:20,494 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:54:20,494 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:54:20,494 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:54:20,494 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:54:20,494 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:54:20,495 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:20,495 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:20,496 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:20,496 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:54:20,496 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:54:20,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:54:20,496 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:20,496 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:54:20,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:20,497 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:20,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:54:20,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:20,497 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:54:20,497 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:54:20,497 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:54:20,497 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:54:20,497 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:54:20,497 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:54:20,498 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:54:20,498 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:54:20,498 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f. 2023-04-16 00:54:20,498 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f. started... 2023-04-16 00:54:20,498 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f. as already flushing 2023-04-16 00:54:20,498 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f. 2023-04-16 00:54:20,498 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:54:20,498 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:54:20,498 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:20,498 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:54:20,498 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:54:20,500 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:20,500 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:20,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:20,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:20,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:20,500 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:20,500 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:54:20,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:20,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:20,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:20,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:20,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:20,502 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:54:20,502 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3c99b4a8[Count = 0] remaining members to acquire global barrier 2023-04-16 00:54:20,502 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:54:20,502 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:20,503 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:20,503 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:20,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:20,503 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:54:20,503 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:54:20,503 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:54:20,503 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:20,503 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:54:20,504 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:20,505 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:54:20,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:20,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:20,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:20,505 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:54:20,505 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:54:20,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:20,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:20,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:20,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:20,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:20,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:20,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:20,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:54:20,508 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:54:20,508 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:54:20,508 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:54:20,508 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:54:20,509 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:54:20,509 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:54:20,509 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:20,509 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34680 deadline: 1681606520507, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:20,509 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:20,509 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34740 deadline: 1681606520507, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:20,509 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:20,509 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34718 deadline: 1681606520507, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:20,512 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:20,512 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34714 deadline: 1681606520508, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:20,517 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:20,517 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:54:20,517 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:20,517 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:20,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:20,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:20,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:20,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:20,517 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:54:20,517 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:20,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:20,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:54:20,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:20,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:20,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:20,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:20,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:20,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:20,519 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:20,519 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:20,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:20,519 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34702 deadline: 1681606520517, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:20,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:20,522 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:20,522 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:20,522 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:20,522 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:20,522 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:20,522 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:20,522 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:20,522 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:54:20,522 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:54:20,522 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:54:20,522 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:54:20,522 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:54:20,522 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:54:20,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:54:20,522 DEBUG [Thread-984] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:54:20,522 DEBUG [Thread-984] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:54:20,522 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:54:20,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:54:20,622 DEBUG [Thread-984] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:54:20,623 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:54:20,625 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:54:20,626 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:54:20,626 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:54:20,626 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:54:20,626 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:54:20,626 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:54:20,626 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:54:20,627 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:20,627 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:20,628 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:20,628 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:54:20,628 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:54:20,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:54:20,628 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:20,628 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:54:20,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:20,629 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:20,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:54:20,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:20,629 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:54:20,629 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:54:20,629 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:54:20,629 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:54:20,629 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:54:20,629 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:54:20,632 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:54:20,632 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:54:20,632 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f. 2023-04-16 00:54:20,632 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f. started... 2023-04-16 00:54:20,632 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f. as already flushing 2023-04-16 00:54:20,632 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f. 2023-04-16 00:54:20,632 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:54:20,632 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:54:20,632 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:20,632 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:54:20,632 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:54:20,634 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:20,634 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:20,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:20,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:20,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:20,634 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:20,634 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:54:20,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:20,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:20,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:20,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:20,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:20,636 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:54:20,636 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:54:20,636 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:20,636 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2b5ba5b5[Count = 0] remaining members to acquire global barrier 2023-04-16 00:54:20,639 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:20,639 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:20,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:20,639 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:54:20,639 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:54:20,639 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:54:20,639 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:20,639 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:54:20,640 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:20,640 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:54:20,640 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:54:20,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:20,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:20,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:20,640 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:54:20,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:20,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:20,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:20,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:20,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:20,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:20,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:20,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:54:20,642 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:54:20,642 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:54:20,643 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:54:20,643 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:54:20,643 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:54:20,643 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:54:20,644 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:20,644 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:20,644 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:20,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:20,644 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:54:20,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:20,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:20,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:20,645 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:54:20,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:54:20,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:20,645 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:20,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:20,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:20,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:20,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:20,647 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:20,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:20,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:20,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:20,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:20,650 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:54:20,650 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:54:20,650 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:54:20,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:54:20,650 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:54:20,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:54:20,651 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:20,651 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:20,651 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:20,651 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:54:20,651 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:54:20,651 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:54:20,651 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:20,651 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:20,651 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:20,651 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:20,651 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:54:20,651 DEBUG [Thread-984] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:54:20,651 DEBUG [Thread-984] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:54:20,674 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=50, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/.tmp/A/d42d49a6ea17498eafcd86ca67f9d886 2023-04-16 00:54:20,704 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=50 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/.tmp/B/a54d08fb83414d0f9473426c7209b4a4 2023-04-16 00:54:20,715 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=50 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/.tmp/C/3d7ec6f512f24453bc3ce42107d348ce 2023-04-16 00:54:20,719 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/.tmp/A/d42d49a6ea17498eafcd86ca67f9d886 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/A/d42d49a6ea17498eafcd86ca67f9d886 2023-04-16 00:54:20,723 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/A/d42d49a6ea17498eafcd86ca67f9d886, entries=350, sequenceid=50, filesize=52.3 K 2023-04-16 00:54:20,725 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/.tmp/B/a54d08fb83414d0f9473426c7209b4a4 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/B/a54d08fb83414d0f9473426c7209b4a4 2023-04-16 00:54:20,730 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/B/a54d08fb83414d0f9473426c7209b4a4, entries=150, sequenceid=50, filesize=11.6 K 2023-04-16 00:54:20,731 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/.tmp/C/3d7ec6f512f24453bc3ce42107d348ce as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/C/3d7ec6f512f24453bc3ce42107d348ce 2023-04-16 00:54:20,735 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/C/3d7ec6f512f24453bc3ce42107d348ce, entries=150, sequenceid=50, filesize=11.6 K 2023-04-16 00:54:20,736 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=140.89 KB/144270 for bcfff279d7c9f90f5c57aef887110e5f in 912ms, sequenceid=50, compaction requested=true 2023-04-16 00:54:20,736 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for bcfff279d7c9f90f5c57aef887110e5f: 2023-04-16 00:54:20,736 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 00:54:20,736 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-16 00:54:20,736 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-16 00:54:20,736 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 00:54:20,736 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 00:54:20,738 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 105002 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-16 00:54:20,738 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HStore(1912): bcfff279d7c9f90f5c57aef887110e5f/A is initiating minor compaction (all files) 2023-04-16 00:54:20,738 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-16 00:54:20,738 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of bcfff279d7c9f90f5c57aef887110e5f/A in TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f. 2023-04-16 00:54:20,738 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HStore(1912): bcfff279d7c9f90f5c57aef887110e5f/B is initiating minor compaction (all files) 2023-04-16 00:54:20,738 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/A/e1d6985772b2459891a09ef7b6f7a71a, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/A/c3bc52b00b004f1ead6c43393355b4f9, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/A/d42d49a6ea17498eafcd86ca67f9d886] into tmpdir=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/.tmp, totalSize=102.5 K 2023-04-16 00:54:20,738 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HRegion(2259): Starting compaction of bcfff279d7c9f90f5c57aef887110e5f/B in TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f. 2023-04-16 00:54:20,738 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/B/3f116ca5aea14ee089ffd2938d1a6e36, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/B/004575861f934c2da4ab487a312b7e5d, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/B/a54d08fb83414d0f9473426c7209b4a4] into tmpdir=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/.tmp, totalSize=34.8 K 2023-04-16 00:54:20,739 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.Compactor(207): Compacting e1d6985772b2459891a09ef7b6f7a71a, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=17, earliestPutTs=1681606457582 2023-04-16 00:54:20,740 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting 3f116ca5aea14ee089ffd2938d1a6e36, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=17, earliestPutTs=1681606457582 2023-04-16 00:54:20,740 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.Compactor(207): Compacting c3bc52b00b004f1ead6c43393355b4f9, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=36, earliestPutTs=1681606457654 2023-04-16 00:54:20,741 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting 004575861f934c2da4ab487a312b7e5d, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=36, earliestPutTs=1681606457654 2023-04-16 00:54:20,741 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.Compactor(207): Compacting d42d49a6ea17498eafcd86ca67f9d886, keycount=350, bloomtype=ROW, size=52.3 K, encoding=NONE, compression=NONE, seqNum=50, earliestPutTs=1681606459811 2023-04-16 00:54:20,741 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting a54d08fb83414d0f9473426c7209b4a4, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=50, earliestPutTs=1681606459814 2023-04-16 00:54:20,750 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] throttle.PressureAwareThroughputController(145): bcfff279d7c9f90f5c57aef887110e5f#B#compaction#136 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-16 00:54:20,751 DEBUG [Thread-984] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:54:20,756 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:54:20,758 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:54:20,763 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:54:20,764 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:54:20,764 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:54:20,764 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:54:20,764 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:54:20,764 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:54:20,764 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:20,764 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:20,767 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] throttle.PressureAwareThroughputController(145): bcfff279d7c9f90f5c57aef887110e5f#A#compaction#135 average throughput is 1.10 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-16 00:54:20,767 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:20,767 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:54:20,767 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:54:20,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:54:20,768 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:20,768 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:54:20,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:20,768 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:20,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:54:20,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:20,769 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:54:20,776 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:54:20,776 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:54:20,776 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:54:20,776 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:54:20,776 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:54:20,788 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:54:20,788 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:54:20,788 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f. 2023-04-16 00:54:20,788 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f. started... 2023-04-16 00:54:20,788 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing bcfff279d7c9f90f5c57aef887110e5f 3/3 column families, dataSize=140.89 KB heapSize=390.84 KB 2023-04-16 00:54:20,789 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bcfff279d7c9f90f5c57aef887110e5f, store=A 2023-04-16 00:54:20,790 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 00:54:20,790 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bcfff279d7c9f90f5c57aef887110e5f, store=B 2023-04-16 00:54:20,790 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 00:54:20,790 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bcfff279d7c9f90f5c57aef887110e5f, store=C 2023-04-16 00:54:20,790 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 00:54:20,802 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/.tmp/B/9d8eae1b34a04e2f9b603066470cb6ca as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/B/9d8eae1b34a04e2f9b603066470cb6ca 2023-04-16 00:54:20,816 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in bcfff279d7c9f90f5c57aef887110e5f/B of bcfff279d7c9f90f5c57aef887110e5f into 9d8eae1b34a04e2f9b603066470cb6ca(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 00:54:20,816 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for bcfff279d7c9f90f5c57aef887110e5f: 2023-04-16 00:54:20,816 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f., storeName=bcfff279d7c9f90f5c57aef887110e5f/B, priority=13, startTime=1681606460736; duration=0sec 2023-04-16 00:54:20,817 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 00:54:20,817 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-16 00:54:20,822 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-16 00:54:20,823 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HStore(1912): bcfff279d7c9f90f5c57aef887110e5f/C is initiating minor compaction (all files) 2023-04-16 00:54:20,823 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HRegion(2259): Starting compaction of bcfff279d7c9f90f5c57aef887110e5f/C in TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f. 2023-04-16 00:54:20,823 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/C/61b84698f9a14cf0869de0b18914632a, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/C/f44186f6c28d4fce89bbc0768d661e1e, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/C/3d7ec6f512f24453bc3ce42107d348ce] into tmpdir=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/.tmp, totalSize=34.8 K 2023-04-16 00:54:20,823 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting 61b84698f9a14cf0869de0b18914632a, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=17, earliestPutTs=1681606457582 2023-04-16 00:54:20,824 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting f44186f6c28d4fce89bbc0768d661e1e, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=36, earliestPutTs=1681606457654 2023-04-16 00:54:20,824 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting 3d7ec6f512f24453bc3ce42107d348ce, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=50, earliestPutTs=1681606459814 2023-04-16 00:54:20,833 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230416b5a9deceffe44e4280134c07b9d47620 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230416b5a9deceffe44e4280134c07b9d47620 2023-04-16 00:54:20,833 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] throttle.PressureAwareThroughputController(145): bcfff279d7c9f90f5c57aef887110e5f#C#compaction#138 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-16 00:54:21,014 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(9158): Flush requested on bcfff279d7c9f90f5c57aef887110e5f 2023-04-16 00:54:21,014 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f. as already flushing 2023-04-16 00:54:21,021 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:21,021 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34740 deadline: 1681606521019, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:21,023 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:21,023 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34714 deadline: 1681606521020, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:21,023 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:21,023 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34680 deadline: 1681606521021, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:21,023 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:21,023 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34718 deadline: 1681606521022, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:21,023 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:21,024 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34702 deadline: 1681606521022, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:21,125 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:21,125 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34740 deadline: 1681606521122, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:21,126 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:21,127 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34714 deadline: 1681606521124, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:21,127 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:21,127 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34680 deadline: 1681606521124, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:21,127 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:21,127 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34718 deadline: 1681606521124, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:21,246 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=85, memsize=49.2 K, hasBloomFilter=true, into tmp file hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/.tmp/A/562893653eb54f86ac7befc2f34095cc 2023-04-16 00:54:21,254 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/.tmp/C/81819213e8eb40c39c93e8431207dc9d as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/C/81819213e8eb40c39c93e8431207dc9d 2023-04-16 00:54:21,261 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in bcfff279d7c9f90f5c57aef887110e5f/C of bcfff279d7c9f90f5c57aef887110e5f into 81819213e8eb40c39c93e8431207dc9d(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 00:54:21,261 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for bcfff279d7c9f90f5c57aef887110e5f: 2023-04-16 00:54:21,261 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f., storeName=bcfff279d7c9f90f5c57aef887110e5f/C, priority=13, startTime=1681606460736; duration=0sec 2023-04-16 00:54:21,261 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 00:54:21,264 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=85 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/.tmp/B/60b85be4841644a385e443b5d1e20625 2023-04-16 00:54:21,329 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:21,329 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34740 deadline: 1681606521327, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:21,329 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:21,329 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34680 deadline: 1681606521328, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:21,329 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:21,329 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34714 deadline: 1681606521329, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:21,329 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:21,329 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34718 deadline: 1681606521329, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:21,630 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:21,631 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34740 deadline: 1681606521630, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:21,631 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:21,631 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34680 deadline: 1681606521630, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:21,634 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:21,634 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34718 deadline: 1681606521632, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:21,636 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:21,636 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34714 deadline: 1681606521632, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:21,637 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/.tmp/A/d94226ccd6774623b5d4f922c8e69e9e as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/A/d94226ccd6774623b5d4f922c8e69e9e 2023-04-16 00:54:21,647 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in bcfff279d7c9f90f5c57aef887110e5f/A of bcfff279d7c9f90f5c57aef887110e5f into d94226ccd6774623b5d4f922c8e69e9e(size=25.2 K), total size for store is 25.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 00:54:21,647 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for bcfff279d7c9f90f5c57aef887110e5f: 2023-04-16 00:54:21,647 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f., storeName=bcfff279d7c9f90f5c57aef887110e5f/A, priority=13, startTime=1681606460736; duration=0sec 2023-04-16 00:54:21,647 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 00:54:21,675 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=85 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/.tmp/C/8eccc915c86748938f90f1b89ad3d918 2023-04-16 00:54:21,681 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/.tmp/A/562893653eb54f86ac7befc2f34095cc as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/A/562893653eb54f86ac7befc2f34095cc 2023-04-16 00:54:21,687 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/A/562893653eb54f86ac7befc2f34095cc, entries=150, sequenceid=85, filesize=25.1 K 2023-04-16 00:54:21,689 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/.tmp/B/60b85be4841644a385e443b5d1e20625 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/B/60b85be4841644a385e443b5d1e20625 2023-04-16 00:54:21,698 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/B/60b85be4841644a385e443b5d1e20625, entries=150, sequenceid=85, filesize=11.6 K 2023-04-16 00:54:21,699 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/.tmp/C/8eccc915c86748938f90f1b89ad3d918 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/C/8eccc915c86748938f90f1b89ad3d918 2023-04-16 00:54:21,715 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/C/8eccc915c86748938f90f1b89ad3d918, entries=150, sequenceid=85, filesize=11.6 K 2023-04-16 00:54:21,716 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~390.80 KB/400176, currentSize=46.96 KB/48090 for bcfff279d7c9f90f5c57aef887110e5f in 928ms, sequenceid=85, compaction requested=false 2023-04-16 00:54:21,716 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for bcfff279d7c9f90f5c57aef887110e5f: 2023-04-16 00:54:21,716 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f. 2023-04-16 00:54:21,716 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:54:21,716 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:54:21,716 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:21,716 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:54:21,716 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:54:21,718 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:21,718 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:21,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:21,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:21,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:21,718 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:21,718 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:54:21,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:21,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:21,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:21,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:21,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:21,720 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:54:21,720 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@47f0d7f8[Count = 0] remaining members to acquire global barrier 2023-04-16 00:54:21,720 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:54:21,720 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:21,721 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:21,722 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:21,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:21,722 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:54:21,722 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:54:21,722 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:54:21,722 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:21,722 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:54:21,723 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:21,723 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:54:21,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:21,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:21,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:21,723 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:54:21,723 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:54:21,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:21,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:21,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:21,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:21,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:21,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:21,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:21,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:54:21,726 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:54:21,726 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:54:21,726 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:54:21,726 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:54:21,726 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:54:21,726 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:54:21,736 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:21,736 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:54:21,736 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:21,736 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:21,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:21,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:21,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:21,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:21,736 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:54:21,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:54:21,737 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:21,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:21,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:21,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:21,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:21,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:21,738 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:21,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:21,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:21,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:21,741 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:21,741 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:21,741 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:21,741 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:21,741 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:21,741 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:54:21,741 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:54:21,741 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:54:21,741 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:54:21,741 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:21,741 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:21,741 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:54:21,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:54:21,741 DEBUG [Thread-984] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:54:21,741 DEBUG [Thread-984] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:54:21,741 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:54:21,742 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:54:21,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:54:21,743 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:54:21,841 DEBUG [Thread-984] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:54:21,842 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:54:21,844 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:54:21,847 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:54:21,847 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:54:21,848 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:54:21,848 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:54:21,848 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:54:21,848 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:54:21,848 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:21,848 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:21,850 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:54:21,850 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:54:21,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:54:21,850 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:21,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:21,850 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:21,850 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:21,851 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:54:21,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:54:21,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:21,851 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:54:21,851 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:54:21,852 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:54:21,852 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:54:21,852 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:54:21,852 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:54:21,852 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:54:21,852 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f. 2023-04-16 00:54:21,852 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f. started... 2023-04-16 00:54:21,852 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing bcfff279d7c9f90f5c57aef887110e5f 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-16 00:54:21,852 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bcfff279d7c9f90f5c57aef887110e5f, store=A 2023-04-16 00:54:21,852 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:54:21,853 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bcfff279d7c9f90f5c57aef887110e5f, store=B 2023-04-16 00:54:21,853 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:54:21,853 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bcfff279d7c9f90f5c57aef887110e5f, store=C 2023-04-16 00:54:21,853 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:54:22,031 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(9158): Flush requested on bcfff279d7c9f90f5c57aef887110e5f 2023-04-16 00:54:22,031 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f. as already flushing 2023-04-16 00:54:22,169 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=459, cellSize=21015, totalHeapSize=47976, min timestamp=1681606462118, max timestamp=1681606462166Num uniques 859; 2023-04-16 00:54:22,169 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=21015, suffix heap size=117064,new segment heap size=47976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=459 2023-04-16 00:54:22,169 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=459, cellSize=21015, totalHeapSize=47976, min timestamp=1681606462118, max timestamp=1681606462166Num uniques 859; 2023-04-16 00:54:22,169 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=459, cellSize=21015, totalHeapSize=47976, min timestamp=1681606462118, max timestamp=1681606462166Num uniques 859; 2023-04-16 00:54:22,169 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=21015, suffix heap size=117064,new segment heap size=47976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=459 2023-04-16 00:54:22,169 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=21015, suffix heap size=117064,new segment heap size=47976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=459 2023-04-16 00:54:22,185 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:22,185 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34702 deadline: 1681606522179, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:22,189 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:22,189 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34740 deadline: 1681606522184, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:22,189 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:22,189 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34680 deadline: 1681606522184, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:22,189 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:22,189 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34718 deadline: 1681606522185, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:22,190 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:22,190 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34714 deadline: 1681606522185, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:22,288 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023041631de60ca1aea4853bbb7eb2179df4428 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041631de60ca1aea4853bbb7eb2179df4428 2023-04-16 00:54:22,291 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:22,291 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34702 deadline: 1681606522287, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:22,291 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:22,291 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34740 deadline: 1681606522290, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:22,291 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:22,291 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34718 deadline: 1681606522291, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:22,292 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:22,292 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34714 deadline: 1681606522291, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:22,292 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:22,292 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34680 deadline: 1681606522291, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:22,299 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=98, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/.tmp/A/323f8a53904c4a2f8e2f130ee9dd9392 2023-04-16 00:54:22,323 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=98 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/.tmp/B/4fe52a20c50b4913b5f13dd3ac7e3bb4 2023-04-16 00:54:22,342 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=98 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/.tmp/C/ee0b7de082fa424ca99f0a881ada0ba5 2023-04-16 00:54:22,346 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/.tmp/A/323f8a53904c4a2f8e2f130ee9dd9392 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/A/323f8a53904c4a2f8e2f130ee9dd9392 2023-04-16 00:54:22,351 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/A/323f8a53904c4a2f8e2f130ee9dd9392, entries=100, sequenceid=98, filesize=18.4 K 2023-04-16 00:54:22,352 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/.tmp/B/4fe52a20c50b4913b5f13dd3ac7e3bb4 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/B/4fe52a20c50b4913b5f13dd3ac7e3bb4 2023-04-16 00:54:22,357 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/B/4fe52a20c50b4913b5f13dd3ac7e3bb4, entries=100, sequenceid=98, filesize=9.3 K 2023-04-16 00:54:22,358 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/.tmp/C/ee0b7de082fa424ca99f0a881ada0ba5 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/C/ee0b7de082fa424ca99f0a881ada0ba5 2023-04-16 00:54:22,364 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/C/ee0b7de082fa424ca99f0a881ada0ba5, entries=100, sequenceid=98, filesize=9.3 K 2023-04-16 00:54:22,365 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=147.60 KB/151140 for bcfff279d7c9f90f5c57aef887110e5f in 513ms, sequenceid=98, compaction requested=true 2023-04-16 00:54:22,365 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for bcfff279d7c9f90f5c57aef887110e5f: 2023-04-16 00:54:22,365 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f. 2023-04-16 00:54:22,365 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:54:22,365 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:54:22,365 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:22,365 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:54:22,365 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:54:22,367 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:22,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:22,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:22,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:22,367 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:22,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:22,367 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:22,367 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:54:22,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:22,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:22,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:22,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:22,371 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:54:22,371 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@56bb8ddb[Count = 0] remaining members to acquire global barrier 2023-04-16 00:54:22,371 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:54:22,371 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:22,373 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:22,373 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:22,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:22,373 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:54:22,374 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:54:22,374 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:54:22,374 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:22,374 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:54:22,375 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:54:22,375 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:22,375 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:54:22,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:22,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:22,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:22,375 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:54:22,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:22,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:22,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:22,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:22,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:22,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:22,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:22,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:54:22,379 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:54:22,379 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:54:22,379 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:54:22,379 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:54:22,379 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:54:22,379 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:54:22,381 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:22,381 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:22,381 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:22,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:22,381 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:54:22,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:22,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:22,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:22,381 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:54:22,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:54:22,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:22,383 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:22,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:22,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:22,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:22,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:22,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:22,384 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:22,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:22,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:22,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:22,391 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:22,391 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:22,391 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:22,391 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:22,391 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:22,391 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:22,391 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:22,391 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:54:22,391 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:54:22,391 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:54:22,391 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:54:22,391 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:54:22,391 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:54:22,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:54:22,392 DEBUG [Thread-984] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:54:22,392 DEBUG [Thread-984] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:54:22,392 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:54:22,392 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:54:22,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:54:22,492 DEBUG [Thread-984] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:54:22,493 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:54:22,495 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:54:22,496 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(9158): Flush requested on bcfff279d7c9f90f5c57aef887110e5f 2023-04-16 00:54:22,496 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing bcfff279d7c9f90f5c57aef887110e5f 3/3 column families, dataSize=154.31 KB heapSize=416.63 KB 2023-04-16 00:54:22,497 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:54:22,497 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:54:22,497 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:54:22,497 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:54:22,497 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:54:22,497 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:54:22,497 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:22,497 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:22,498 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bcfff279d7c9f90f5c57aef887110e5f, store=A 2023-04-16 00:54:22,498 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 00:54:22,498 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bcfff279d7c9f90f5c57aef887110e5f, store=B 2023-04-16 00:54:22,498 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 00:54:22,498 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bcfff279d7c9f90f5c57aef887110e5f, store=C 2023-04-16 00:54:22,498 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 00:54:22,499 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:54:22,499 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:22,499 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:54:22,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:54:22,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:22,500 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:22,500 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:22,500 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:54:22,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:54:22,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:22,500 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:54:22,501 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:54:22,501 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:54:22,501 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:54:22,501 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:54:22,501 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:54:22,503 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:54:22,503 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:54:22,503 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f. 2023-04-16 00:54:22,503 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f. started... 2023-04-16 00:54:22,503 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f. as already flushing 2023-04-16 00:54:22,503 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f. 2023-04-16 00:54:22,503 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:54:22,503 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:54:22,503 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:22,503 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:54:22,503 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:54:22,505 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:22,505 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34740 deadline: 1681606522498, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:22,505 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:22,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:22,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:22,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:22,505 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:22,505 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34702 deadline: 1681606522501, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:22,506 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:22,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:22,506 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:22,506 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34718 deadline: 1681606522502, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:22,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:22,506 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:22,506 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:54:22,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:22,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:22,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:22,507 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:54:22,507 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@27f65f50[Count = 0] remaining members to acquire global barrier 2023-04-16 00:54:22,507 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:54:22,507 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:22,508 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:22,509 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:22,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:22,509 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:54:22,509 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:54:22,509 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:54:22,509 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:22,509 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:54:22,512 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:22,512 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34714 deadline: 1681606522506, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:22,512 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:22,512 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34680 deadline: 1681606522506, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:22,512 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:54:22,512 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:54:22,512 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:54:22,514 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:22,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:22,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:22,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:22,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:22,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:22,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:22,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:22,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:22,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:22,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:22,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:54:22,517 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:54:22,517 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:54:22,517 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:54:22,517 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:54:22,517 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:54:22,517 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:54:22,521 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:22,521 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:54:22,521 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:22,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:22,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:22,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:22,521 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:22,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:22,521 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:54:22,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:54:22,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:22,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:22,522 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:22,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:22,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:22,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:22,523 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:22,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:22,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:22,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:22,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:22,526 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:54:22,526 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:54:22,526 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:54:22,526 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:54:22,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:54:22,526 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:54:22,526 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:54:22,526 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:22,526 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:22,526 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:22,526 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:22,526 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:22,526 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:22,526 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:22,526 DEBUG [Thread-984] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:54:22,527 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:54:22,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:54:22,527 DEBUG [Thread-984] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:54:22,527 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:54:22,540 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230416a0eaafc55a8443a381bb0fff82d118ea to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230416a0eaafc55a8443a381bb0fff82d118ea 2023-04-16 00:54:22,550 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=134, memsize=55.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/.tmp/A/7ec2d64efa9d42c88357a6e4c185ee0c 2023-04-16 00:54:22,578 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=55.91 KB at sequenceid=134 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/.tmp/B/84950f4f4c394a79867b48b57fbfbb7a 2023-04-16 00:54:22,598 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=55.91 KB at sequenceid=134 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/.tmp/C/1426976e5f1b487c9b2a3b33ba577825 2023-04-16 00:54:22,605 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/.tmp/A/7ec2d64efa9d42c88357a6e4c185ee0c as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/A/7ec2d64efa9d42c88357a6e4c185ee0c 2023-04-16 00:54:22,609 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:22,609 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34740 deadline: 1681606522606, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:22,609 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:22,609 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34702 deadline: 1681606522606, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:22,609 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:22,609 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34718 deadline: 1681606522607, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:22,610 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/A/7ec2d64efa9d42c88357a6e4c185ee0c, entries=150, sequenceid=134, filesize=25.3 K 2023-04-16 00:54:22,611 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/.tmp/B/84950f4f4c394a79867b48b57fbfbb7a as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/B/84950f4f4c394a79867b48b57fbfbb7a 2023-04-16 00:54:22,616 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/B/84950f4f4c394a79867b48b57fbfbb7a, entries=150, sequenceid=134, filesize=11.8 K 2023-04-16 00:54:22,617 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/.tmp/C/1426976e5f1b487c9b2a3b33ba577825 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/C/1426976e5f1b487c9b2a3b33ba577825 2023-04-16 00:54:22,617 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:22,617 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34680 deadline: 1681606522614, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:22,617 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:22,617 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34714 deadline: 1681606522614, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:22,625 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/C/1426976e5f1b487c9b2a3b33ba577825, entries=150, sequenceid=134, filesize=11.8 K 2023-04-16 00:54:22,628 DEBUG [Thread-984] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:54:22,628 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~167.72 KB/171750, heapSize ~456.42 KB/467376, currentSize=26.84 KB/27480 for bcfff279d7c9f90f5c57aef887110e5f in 132ms, sequenceid=134, compaction requested=true 2023-04-16 00:54:22,628 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for bcfff279d7c9f90f5c57aef887110e5f: 2023-04-16 00:54:22,628 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:54:22,629 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 00:54:22,629 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-16 00:54:22,629 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-16 00:54:22,629 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 00:54:22,629 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 00:54:22,630 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:54:22,631 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 45453 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-16 00:54:22,631 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HStore(1912): bcfff279d7c9f90f5c57aef887110e5f/B is initiating minor compaction (all files) 2023-04-16 00:54:22,631 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HRegion(2259): Starting compaction of bcfff279d7c9f90f5c57aef887110e5f/B in TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f. 2023-04-16 00:54:22,631 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/B/9d8eae1b34a04e2f9b603066470cb6ca, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/B/60b85be4841644a385e443b5d1e20625, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/B/4fe52a20c50b4913b5f13dd3ac7e3bb4, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/B/84950f4f4c394a79867b48b57fbfbb7a] into tmpdir=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/.tmp, totalSize=44.4 K 2023-04-16 00:54:22,632 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 96289 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-16 00:54:22,632 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting 9d8eae1b34a04e2f9b603066470cb6ca, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=50, earliestPutTs=1681606459814 2023-04-16 00:54:22,632 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HStore(1912): bcfff279d7c9f90f5c57aef887110e5f/A is initiating minor compaction (all files) 2023-04-16 00:54:22,632 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of bcfff279d7c9f90f5c57aef887110e5f/A in TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f. 2023-04-16 00:54:22,632 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/A/d94226ccd6774623b5d4f922c8e69e9e, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/A/562893653eb54f86ac7befc2f34095cc, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/A/323f8a53904c4a2f8e2f130ee9dd9392, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/A/7ec2d64efa9d42c88357a6e4c185ee0c] into tmpdir=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/.tmp, totalSize=94.0 K 2023-04-16 00:54:22,634 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting 60b85be4841644a385e443b5d1e20625, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=85, earliestPutTs=1681606459891 2023-04-16 00:54:22,634 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.Compactor(207): Compacting d94226ccd6774623b5d4f922c8e69e9e, keycount=150, bloomtype=ROW, size=25.2 K, encoding=NONE, compression=NONE, seqNum=50, earliestPutTs=1681606459814 2023-04-16 00:54:22,634 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting 4fe52a20c50b4913b5f13dd3ac7e3bb4, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=98, earliestPutTs=1681606461019 2023-04-16 00:54:22,634 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.Compactor(207): Compacting 562893653eb54f86ac7befc2f34095cc, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=85, earliestPutTs=1681606459891 2023-04-16 00:54:22,635 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting 84950f4f4c394a79867b48b57fbfbb7a, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=134, earliestPutTs=1681606462495 2023-04-16 00:54:22,635 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.Compactor(207): Compacting 323f8a53904c4a2f8e2f130ee9dd9392, keycount=100, bloomtype=ROW, size=18.4 K, encoding=NONE, compression=NONE, seqNum=98, earliestPutTs=1681606461019 2023-04-16 00:54:22,635 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.Compactor(207): Compacting 7ec2d64efa9d42c88357a6e4c185ee0c, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=134, earliestPutTs=1681606462495 2023-04-16 00:54:22,636 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:54:22,642 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:54:22,643 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:54:22,643 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:54:22,643 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:54:22,643 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:54:22,646 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:22,646 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:22,648 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:54:22,648 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:22,648 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:54:22,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:54:22,648 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:22,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:22,648 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:54:22,649 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:22,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:54:22,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:22,649 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:54:22,650 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:54:22,650 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:54:22,650 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:54:22,650 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:54:22,650 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:54:22,650 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:54:22,650 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:54:22,651 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f. 2023-04-16 00:54:22,651 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f. started... 2023-04-16 00:54:22,651 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing bcfff279d7c9f90f5c57aef887110e5f 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-04-16 00:54:22,651 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bcfff279d7c9f90f5c57aef887110e5f, store=A 2023-04-16 00:54:22,651 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:54:22,651 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bcfff279d7c9f90f5c57aef887110e5f, store=B 2023-04-16 00:54:22,651 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:54:22,651 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bcfff279d7c9f90f5c57aef887110e5f, store=C 2023-04-16 00:54:22,651 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:54:22,656 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] throttle.PressureAwareThroughputController(145): bcfff279d7c9f90f5c57aef887110e5f#B#compaction#147 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-16 00:54:22,682 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] throttle.PressureAwareThroughputController(145): bcfff279d7c9f90f5c57aef887110e5f#A#compaction#148 average throughput is 0.82 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-16 00:54:22,723 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304166a7f1e1d967940e097800985bd42141a to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304166a7f1e1d967940e097800985bd42141a 2023-04-16 00:54:22,725 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/.tmp/B/8e66594f3e484d3eab1c157a1b7e145d as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/B/8e66594f3e484d3eab1c157a1b7e145d 2023-04-16 00:54:22,734 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in bcfff279d7c9f90f5c57aef887110e5f/B of bcfff279d7c9f90f5c57aef887110e5f into 8e66594f3e484d3eab1c157a1b7e145d(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 00:54:22,735 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for bcfff279d7c9f90f5c57aef887110e5f: 2023-04-16 00:54:22,735 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f., storeName=bcfff279d7c9f90f5c57aef887110e5f/B, priority=12, startTime=1681606462629; duration=0sec 2023-04-16 00:54:22,735 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 00:54:22,735 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-16 00:54:22,735 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=141, memsize=8.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/.tmp/A/5d3dcad2480e49d486ddb3a48152bdf2 2023-04-16 00:54:22,739 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 45453 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-16 00:54:22,739 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HStore(1912): bcfff279d7c9f90f5c57aef887110e5f/C is initiating minor compaction (all files) 2023-04-16 00:54:22,739 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HRegion(2259): Starting compaction of bcfff279d7c9f90f5c57aef887110e5f/C in TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f. 2023-04-16 00:54:22,739 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/C/81819213e8eb40c39c93e8431207dc9d, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/C/8eccc915c86748938f90f1b89ad3d918, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/C/ee0b7de082fa424ca99f0a881ada0ba5, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/C/1426976e5f1b487c9b2a3b33ba577825] into tmpdir=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/.tmp, totalSize=44.4 K 2023-04-16 00:54:22,739 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting 81819213e8eb40c39c93e8431207dc9d, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=50, earliestPutTs=1681606459814 2023-04-16 00:54:22,740 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting 8eccc915c86748938f90f1b89ad3d918, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=85, earliestPutTs=1681606459891 2023-04-16 00:54:22,740 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting ee0b7de082fa424ca99f0a881ada0ba5, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=98, earliestPutTs=1681606461019 2023-04-16 00:54:22,741 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting 1426976e5f1b487c9b2a3b33ba577825, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=134, earliestPutTs=1681606462495 2023-04-16 00:54:22,751 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] throttle.PressureAwareThroughputController(145): bcfff279d7c9f90f5c57aef887110e5f#C#compaction#151 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-16 00:54:22,774 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/.tmp/C/b57b50aacd47434280ba1d9b02ca28fb as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/C/b57b50aacd47434280ba1d9b02ca28fb 2023-04-16 00:54:22,779 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in bcfff279d7c9f90f5c57aef887110e5f/C of bcfff279d7c9f90f5c57aef887110e5f into b57b50aacd47434280ba1d9b02ca28fb(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 00:54:22,779 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for bcfff279d7c9f90f5c57aef887110e5f: 2023-04-16 00:54:22,779 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f., storeName=bcfff279d7c9f90f5c57aef887110e5f/C, priority=12, startTime=1681606462629; duration=0sec 2023-04-16 00:54:22,779 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 00:54:22,814 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(9158): Flush requested on bcfff279d7c9f90f5c57aef887110e5f 2023-04-16 00:54:22,815 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f. as already flushing 2023-04-16 00:54:22,852 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14144, totalHeapSize=32376, min timestamp=1681606462821, max timestamp=1681606462850Num uniques 859; 2023-04-16 00:54:22,852 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=14144, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-16 00:54:22,852 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14144, totalHeapSize=32376, min timestamp=1681606462821, max timestamp=1681606462850Num uniques 859; 2023-04-16 00:54:22,852 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=14144, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-16 00:54:22,852 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14144, totalHeapSize=32376, min timestamp=1681606462821, max timestamp=1681606462850Num uniques 859; 2023-04-16 00:54:22,852 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=14144, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-16 00:54:22,874 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=368, cellSize=16848, totalHeapSize=38512, min timestamp=1681606462857, max timestamp=1681606462871Num uniques 1168; 2023-04-16 00:54:22,874 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=368, cellSize=16848, totalHeapSize=38512, min timestamp=1681606462857, max timestamp=1681606462871Num uniques 1168; 2023-04-16 00:54:22,874 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=53488, new segment data size=16848, suffix heap size=149440,new segment heap size=38512 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1168, new segment cells count=368 2023-04-16 00:54:22,874 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=53488, new segment data size=16848, suffix heap size=149440,new segment heap size=38512 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1168, new segment cells count=368 2023-04-16 00:54:22,874 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=368, cellSize=16848, totalHeapSize=38512, min timestamp=1681606462857, max timestamp=1681606462871Num uniques 1168; 2023-04-16 00:54:22,875 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=53488, new segment data size=16848, suffix heap size=149440,new segment heap size=38512 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1168, new segment cells count=368 2023-04-16 00:54:22,893 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:22,893 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34680 deadline: 1681606522889, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:22,893 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:22,893 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34702 deadline: 1681606522890, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:22,894 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:22,894 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34718 deadline: 1681606522891, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:22,894 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:22,894 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34740 deadline: 1681606522892, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:22,894 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:22,894 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34714 deadline: 1681606522893, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:22,996 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:22,996 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34680 deadline: 1681606522994, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:22,996 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:22,996 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34718 deadline: 1681606522995, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:22,996 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:22,996 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34702 deadline: 1681606522995, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:22,997 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:22,997 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34740 deadline: 1681606522995, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:22,997 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:22,997 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34714 deadline: 1681606522995, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:23,149 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/.tmp/A/ccfd106fb798442d87f12fc11d53a8e5 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/A/ccfd106fb798442d87f12fc11d53a8e5 2023-04-16 00:54:23,154 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in bcfff279d7c9f90f5c57aef887110e5f/A of bcfff279d7c9f90f5c57aef887110e5f into ccfd106fb798442d87f12fc11d53a8e5(size=25.5 K), total size for store is 25.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 00:54:23,154 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for bcfff279d7c9f90f5c57aef887110e5f: 2023-04-16 00:54:23,154 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f., storeName=bcfff279d7c9f90f5c57aef887110e5f/A, priority=12, startTime=1681606462628; duration=0sec 2023-04-16 00:54:23,154 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 00:54:23,162 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=141 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/.tmp/B/f02f9a14b5bb4aacaad5dc979ad36280 2023-04-16 00:54:23,200 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:23,201 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34680 deadline: 1681606523197, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:23,201 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:23,201 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34714 deadline: 1681606523198, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:23,201 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:23,201 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34718 deadline: 1681606523198, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:23,201 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:23,201 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34702 deadline: 1681606523198, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:23,202 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:23,202 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34740 deadline: 1681606523199, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:23,504 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:23,504 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34714 deadline: 1681606523503, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:23,505 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:23,505 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:23,505 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34680 deadline: 1681606523503, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:23,505 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34718 deadline: 1681606523503, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:23,505 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:23,505 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:23,505 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34740 deadline: 1681606523503, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:23,505 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34702 deadline: 1681606523503, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:23,573 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=141 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/.tmp/C/d499851602f249cf8e595a65615ad78e 2023-04-16 00:54:23,577 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/.tmp/A/5d3dcad2480e49d486ddb3a48152bdf2 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/A/5d3dcad2480e49d486ddb3a48152bdf2 2023-04-16 00:54:23,581 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/A/5d3dcad2480e49d486ddb3a48152bdf2, entries=150, sequenceid=141, filesize=25.3 K 2023-04-16 00:54:23,582 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/.tmp/B/f02f9a14b5bb4aacaad5dc979ad36280 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/B/f02f9a14b5bb4aacaad5dc979ad36280 2023-04-16 00:54:23,590 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/B/f02f9a14b5bb4aacaad5dc979ad36280, entries=150, sequenceid=141, filesize=11.8 K 2023-04-16 00:54:23,591 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/.tmp/C/d499851602f249cf8e595a65615ad78e as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/C/d499851602f249cf8e595a65615ad78e 2023-04-16 00:54:23,595 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/C/d499851602f249cf8e595a65615ad78e, entries=150, sequenceid=141, filesize=11.8 K 2023-04-16 00:54:23,596 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=161.02 KB/164880 for bcfff279d7c9f90f5c57aef887110e5f in 945ms, sequenceid=141, compaction requested=false 2023-04-16 00:54:23,596 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for bcfff279d7c9f90f5c57aef887110e5f: 2023-04-16 00:54:23,596 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f. 2023-04-16 00:54:23,596 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:54:23,596 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:54:23,596 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:23,600 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:54:23,600 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:54:23,601 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:23,601 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:23,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:23,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:23,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:23,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:23,602 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:23,602 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:54:23,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:23,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:23,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:23,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:23,604 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:54:23,604 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@51f63016[Count = 0] remaining members to acquire global barrier 2023-04-16 00:54:23,604 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:54:23,604 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:23,605 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:23,605 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:23,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:23,605 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:54:23,605 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:54:23,605 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:54:23,606 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:23,606 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:54:23,607 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:54:23,607 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:54:23,607 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:23,607 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:54:23,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:23,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:23,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:23,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:23,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:23,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:23,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:23,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:23,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:23,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:23,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:54:23,612 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:54:23,612 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:54:23,612 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:54:23,612 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:54:23,612 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:54:23,612 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:54:23,615 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:23,615 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:23,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:23,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:23,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:23,615 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:23,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:23,615 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:54:23,616 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:54:23,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:54:23,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:23,616 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:23,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:23,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:23,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:23,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:23,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:23,622 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:23,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:23,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:23,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:23,626 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:23,626 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:23,626 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:23,626 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:23,626 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:23,626 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:54:23,626 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:54:23,626 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:23,626 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:54:23,626 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:54:23,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:54:23,626 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:54:23,626 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:54:23,626 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:54:23,626 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:23,626 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:54:23,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:54:23,626 DEBUG [Thread-984] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:54:23,626 DEBUG [Thread-984] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:54:23,727 DEBUG [Thread-984] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:54:23,727 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:54:23,729 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:54:23,730 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:54:23,730 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:54:23,730 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:54:23,730 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:54:23,730 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:54:23,730 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:54:23,731 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:23,731 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:23,733 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:54:23,733 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:23,733 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:54:23,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:54:23,733 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:23,733 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:54:23,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:23,734 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:23,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:54:23,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:23,734 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:54:23,734 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:54:23,734 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:54:23,734 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:54:23,734 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:54:23,734 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:54:23,735 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:54:23,735 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:54:23,735 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f. 2023-04-16 00:54:23,735 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f. started... 2023-04-16 00:54:23,735 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing bcfff279d7c9f90f5c57aef887110e5f 3/3 column families, dataSize=161.02 KB heapSize=445.08 KB 2023-04-16 00:54:23,735 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bcfff279d7c9f90f5c57aef887110e5f, store=A 2023-04-16 00:54:23,735 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 00:54:23,735 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bcfff279d7c9f90f5c57aef887110e5f, store=B 2023-04-16 00:54:23,735 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 00:54:23,735 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bcfff279d7c9f90f5c57aef887110e5f, store=C 2023-04-16 00:54:23,735 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 00:54:24,010 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f. as already flushing 2023-04-16 00:54:24,010 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(9158): Flush requested on bcfff279d7c9f90f5c57aef887110e5f 2023-04-16 00:54:24,013 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:24,013 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34740 deadline: 1681606524010, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:24,013 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:24,013 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34718 deadline: 1681606524010, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:24,014 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:24,014 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34702 deadline: 1681606524011, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:24,015 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:24,015 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34714 deadline: 1681606524012, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:24,016 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:24,016 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34680 deadline: 1681606524013, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:24,116 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:24,116 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34718 deadline: 1681606524114, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:24,116 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:24,116 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34702 deadline: 1681606524114, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:24,117 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:24,117 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34714 deadline: 1681606524115, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:24,118 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:24,119 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34680 deadline: 1681606524117, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:24,150 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230416628c463dccfc4a7c980bd82f6622039f to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230416628c463dccfc4a7c980bd82f6622039f 2023-04-16 00:54:24,319 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:24,320 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34702 deadline: 1681606524317, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:24,320 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:24,320 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34718 deadline: 1681606524318, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:24,320 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:24,320 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34714 deadline: 1681606524319, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:24,320 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:24,320 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34680 deadline: 1681606524320, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:24,555 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=198, memsize=53.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/.tmp/A/430045034cd042ee8842b99fce78fad7 2023-04-16 00:54:24,623 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:24,623 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34702 deadline: 1681606524621, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:24,623 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:24,623 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34680 deadline: 1681606524621, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:24,623 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:24,623 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34714 deadline: 1681606524622, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:24,624 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:24,624 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34718 deadline: 1681606524623, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:24,976 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=53.67 KB at sequenceid=198 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/.tmp/B/05ae3848c7b54920b331066dbd534638 2023-04-16 00:54:25,017 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:25,017 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34740 deadline: 1681606525016, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:25,125 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:25,125 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34680 deadline: 1681606525124, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:25,128 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:25,128 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34718 deadline: 1681606525127, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:25,128 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:25,128 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34702 deadline: 1681606525128, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:25,129 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:25,129 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34714 deadline: 1681606525128, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:25,408 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=53.67 KB at sequenceid=198 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/.tmp/C/f537bd4395704989b5500cab8788532e 2023-04-16 00:54:25,412 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/.tmp/A/430045034cd042ee8842b99fce78fad7 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/A/430045034cd042ee8842b99fce78fad7 2023-04-16 00:54:25,416 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/A/430045034cd042ee8842b99fce78fad7, entries=150, sequenceid=198, filesize=25.3 K 2023-04-16 00:54:25,417 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/.tmp/B/05ae3848c7b54920b331066dbd534638 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/B/05ae3848c7b54920b331066dbd534638 2023-04-16 00:54:25,421 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/B/05ae3848c7b54920b331066dbd534638, entries=150, sequenceid=198, filesize=11.8 K 2023-04-16 00:54:25,422 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/.tmp/C/f537bd4395704989b5500cab8788532e as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/C/f537bd4395704989b5500cab8788532e 2023-04-16 00:54:25,426 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/C/f537bd4395704989b5500cab8788532e, entries=150, sequenceid=198, filesize=11.8 K 2023-04-16 00:54:25,427 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~161.02 KB/164880, heapSize ~445.03 KB/455712, currentSize=26.84 KB/27480 for bcfff279d7c9f90f5c57aef887110e5f in 1692ms, sequenceid=198, compaction requested=true 2023-04-16 00:54:25,427 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for bcfff279d7c9f90f5c57aef887110e5f: 2023-04-16 00:54:25,427 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f. 2023-04-16 00:54:25,427 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:54:25,427 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:54:25,427 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:25,427 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:54:25,427 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:54:25,429 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:25,429 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:25,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:25,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:25,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:25,429 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:25,429 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:54:25,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:25,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:25,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:25,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:25,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:25,431 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:54:25,431 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@107872c7[Count = 0] remaining members to acquire global barrier 2023-04-16 00:54:25,431 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:54:25,431 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:25,435 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:25,435 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:25,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:25,435 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:54:25,435 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:54:25,435 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:25,435 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:54:25,435 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:54:25,437 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:25,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:25,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:25,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:25,437 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:54:25,437 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:54:25,437 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:54:25,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:25,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:25,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:25,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:25,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:25,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:25,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:25,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:54:25,439 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:54:25,439 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:54:25,440 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:54:25,440 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:54:25,440 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:54:25,440 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:54:25,441 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:25,441 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:25,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:25,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:25,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:25,441 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:25,441 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:54:25,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:25,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:25,442 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:25,442 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:54:25,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:54:25,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:25,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:25,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:25,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:25,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:25,444 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:25,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:25,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:25,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:25,454 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:54:25,454 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:54:25,454 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:25,454 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:54:25,454 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:54:25,454 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:54:25,454 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:54:25,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:54:25,454 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:25,454 DEBUG [Thread-984] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:54:25,454 DEBUG [Thread-984] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:54:25,454 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:25,454 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:25,454 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:25,455 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:25,455 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:25,455 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:54:25,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:54:25,554 DEBUG [Thread-984] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:54:25,555 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:54:25,557 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:54:25,559 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:54:25,559 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:54:25,559 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:54:25,559 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:54:25,560 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:54:25,560 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:54:25,560 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:25,560 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:25,561 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:54:25,561 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:25,561 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:54:25,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:54:25,562 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:25,562 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:54:25,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:25,562 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:25,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:54:25,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:25,563 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:54:25,564 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:54:25,564 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:54:25,564 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:54:25,564 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:54:25,564 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:54:25,564 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:54:25,564 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:54:25,565 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f. 2023-04-16 00:54:25,565 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f. started... 2023-04-16 00:54:25,565 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing bcfff279d7c9f90f5c57aef887110e5f 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-04-16 00:54:25,565 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bcfff279d7c9f90f5c57aef887110e5f, store=A 2023-04-16 00:54:25,565 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:54:25,565 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bcfff279d7c9f90f5c57aef887110e5f, store=B 2023-04-16 00:54:25,565 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:54:25,565 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bcfff279d7c9f90f5c57aef887110e5f, store=C 2023-04-16 00:54:25,565 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:54:25,609 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230416a1d3939c9b71438a952377e0262c549e to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230416a1d3939c9b71438a952377e0262c549e 2023-04-16 00:54:25,618 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=205, memsize=8.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/.tmp/A/7e589a35f7264ea482484f88a05d763c 2023-04-16 00:54:25,639 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=205 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/.tmp/B/92aca36b949c48efbb8ad2d0834d0869 2023-04-16 00:54:25,659 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=205 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/.tmp/C/f5360c0edde043ab9f9d1b3a2ee9b3ad 2023-04-16 00:54:25,664 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/.tmp/A/7e589a35f7264ea482484f88a05d763c as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/A/7e589a35f7264ea482484f88a05d763c 2023-04-16 00:54:25,668 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/A/7e589a35f7264ea482484f88a05d763c, entries=100, sequenceid=205, filesize=18.5 K 2023-04-16 00:54:25,669 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/.tmp/B/92aca36b949c48efbb8ad2d0834d0869 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/B/92aca36b949c48efbb8ad2d0834d0869 2023-04-16 00:54:25,674 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/B/92aca36b949c48efbb8ad2d0834d0869, entries=100, sequenceid=205, filesize=9.4 K 2023-04-16 00:54:25,675 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/.tmp/C/f5360c0edde043ab9f9d1b3a2ee9b3ad as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/C/f5360c0edde043ab9f9d1b3a2ee9b3ad 2023-04-16 00:54:25,679 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/C/f5360c0edde043ab9f9d1b3a2ee9b3ad, entries=100, sequenceid=205, filesize=9.4 K 2023-04-16 00:54:25,680 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=0 B/0 for bcfff279d7c9f90f5c57aef887110e5f in 115ms, sequenceid=205, compaction requested=true 2023-04-16 00:54:25,680 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for bcfff279d7c9f90f5c57aef887110e5f: 2023-04-16 00:54:25,680 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f. 2023-04-16 00:54:25,680 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:54:25,680 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:54:25,680 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:25,680 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:54:25,680 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:54:25,682 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:25,682 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:25,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:25,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:25,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:25,682 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:25,682 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:54:25,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:25,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:25,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:25,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:25,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:25,684 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:54:25,684 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1aa7f046[Count = 0] remaining members to acquire global barrier 2023-04-16 00:54:25,684 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:54:25,685 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:25,686 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:25,686 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:25,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:25,686 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:54:25,686 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:54:25,686 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:54:25,686 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:25,686 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:54:25,688 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:54:25,688 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:25,688 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:54:25,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:25,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:25,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:25,689 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:54:25,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:25,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:25,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:25,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:25,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:25,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:25,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:25,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:54:25,692 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:54:25,692 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:54:25,692 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:54:25,692 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:54:25,692 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:54:25,692 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:54:25,693 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:25,694 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:54:25,694 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:25,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:25,694 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:25,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:25,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:25,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:25,694 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:54:25,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:54:25,694 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:25,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:25,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:25,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:25,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:25,696 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:25,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:25,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:25,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:25,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:25,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:25,699 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:54:25,699 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:54:25,699 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:54:25,699 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:54:25,699 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:54:25,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:54:25,699 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:25,700 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:25,700 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:25,699 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:54:25,700 DEBUG [Thread-984] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:54:25,700 DEBUG [Thread-984] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:54:25,700 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:25,700 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:54:25,700 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:54:25,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:54:25,700 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:25,700 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:25,700 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:25,800 DEBUG [Thread-984] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:54:25,801 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:54:25,802 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:54:25,804 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:54:25,804 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:54:25,804 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:54:25,804 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:54:25,805 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:54:25,805 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:54:25,805 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:25,805 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:25,806 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:25,806 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:54:25,806 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:54:25,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:54:25,807 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:25,807 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:54:25,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:25,807 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:25,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:54:25,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:25,808 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:54:25,808 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:54:25,808 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:54:25,808 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:54:25,808 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:54:25,808 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:54:25,808 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:54:25,809 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:54:25,809 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f. 2023-04-16 00:54:25,809 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f. started... 2023-04-16 00:54:25,809 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for bcfff279d7c9f90f5c57aef887110e5f: 2023-04-16 00:54:25,809 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f. 2023-04-16 00:54:25,809 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:54:25,809 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:54:25,809 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:25,809 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:54:25,809 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:54:25,812 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:25,812 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:25,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:25,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:25,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:25,812 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:25,812 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:54:25,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:25,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:25,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:25,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:25,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:25,814 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:54:25,814 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:54:25,814 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:25,814 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@fbe209d[Count = 0] remaining members to acquire global barrier 2023-04-16 00:54:25,815 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:25,815 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:25,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:25,816 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:54:25,816 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:54:25,816 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:54:25,816 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:25,816 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:54:25,817 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:25,817 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:54:25,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:25,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:25,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:25,817 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:54:25,817 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:54:25,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:25,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:25,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:25,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:25,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:25,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:25,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:25,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:54:25,820 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:54:25,820 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:54:25,820 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:54:25,820 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:54:25,820 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:54:25,820 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:54:25,822 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:25,822 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:25,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:25,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:25,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:25,822 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:25,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:25,822 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:54:25,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:25,822 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:54:25,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:54:25,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:25,823 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:25,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:25,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:25,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:25,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:25,824 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:25,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:25,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:25,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:25,828 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:25,828 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:54:25,828 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:54:25,828 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:25,828 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:54:25,828 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:54:25,828 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:54:25,828 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:54:25,828 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:54:25,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:54:25,828 DEBUG [Thread-984] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:54:25,828 DEBUG [Thread-984] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:54:25,828 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:25,828 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:25,828 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:25,828 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:25,828 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:25,829 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:54:25,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:54:25,928 DEBUG [Thread-984] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:54:25,929 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:54:25,931 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:54:25,933 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:54:25,933 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:54:25,933 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:54:25,933 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:54:25,933 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:54:25,933 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:54:25,934 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:25,934 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:25,935 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:25,935 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:54:25,935 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:54:25,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:54:25,936 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:25,936 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:54:25,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:25,936 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:25,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:54:25,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:25,937 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:54:25,937 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:54:25,937 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:54:25,937 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:54:25,937 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:54:25,937 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:54:25,938 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:54:25,938 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:54:25,938 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f. 2023-04-16 00:54:25,938 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f. started... 2023-04-16 00:54:25,938 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for bcfff279d7c9f90f5c57aef887110e5f: 2023-04-16 00:54:25,938 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f. 2023-04-16 00:54:25,938 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:54:25,938 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:54:25,938 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:25,938 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:54:25,938 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:54:25,940 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:25,941 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:25,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:25,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:25,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:25,941 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:25,941 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:54:25,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:25,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:25,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:25,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:25,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:25,943 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:54:25,943 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:54:25,943 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:25,943 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6fa8d7fc[Count = 0] remaining members to acquire global barrier 2023-04-16 00:54:25,944 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:25,944 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:25,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:25,944 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:54:25,944 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:54:25,944 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:54:25,945 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:25,945 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:54:25,947 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:54:25,947 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:25,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:25,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:25,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:25,947 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:54:25,948 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:54:25,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:25,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:25,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:25,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:25,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:25,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:25,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:25,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:54:25,950 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:54:25,950 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:54:25,950 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:54:25,950 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:54:25,950 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:54:25,951 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:54:25,952 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:25,952 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:25,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:25,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:25,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:25,952 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:25,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:25,952 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:54:25,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:25,953 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:54:25,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:54:25,953 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:25,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:25,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:25,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:25,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:25,954 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:25,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:25,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:25,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:25,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:25,957 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:25,957 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:25,957 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:25,957 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:54:25,957 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:54:25,957 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:54:25,957 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:54:25,957 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:54:25,957 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:25,957 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:25,957 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:25,957 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:25,957 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:54:25,958 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:54:25,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:54:25,958 DEBUG [Thread-984] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:54:25,958 DEBUG [Thread-984] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:54:25,958 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:54:25,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:54:26,058 DEBUG [Thread-984] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:54:26,059 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:54:26,060 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:54:26,062 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:54:26,062 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:54:26,062 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:54:26,062 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:54:26,062 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:54:26,062 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:54:26,063 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:26,063 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:26,064 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:26,064 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:54:26,064 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:54:26,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:54:26,065 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:26,065 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:54:26,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:26,065 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:26,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:54:26,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:26,066 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:54:26,066 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:54:26,066 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:54:26,066 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:54:26,067 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:54:26,067 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:54:26,067 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:54:26,067 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:54:26,067 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f. 2023-04-16 00:54:26,067 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f. started... 2023-04-16 00:54:26,068 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for bcfff279d7c9f90f5c57aef887110e5f: 2023-04-16 00:54:26,068 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f. 2023-04-16 00:54:26,068 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:54:26,068 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:54:26,068 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:26,068 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:54:26,068 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:54:26,073 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:26,073 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:26,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:26,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:26,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:26,074 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:26,074 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:54:26,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:26,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:26,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:26,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:26,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:26,075 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:54:26,075 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3c957e84[Count = 0] remaining members to acquire global barrier 2023-04-16 00:54:26,075 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:54:26,075 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:26,077 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:26,077 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:26,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:26,078 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:54:26,078 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:54:26,078 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:54:26,078 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:26,078 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:54:26,079 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:26,079 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:54:26,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:26,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:26,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:26,079 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:54:26,080 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:54:26,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:26,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:26,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:26,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:26,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:26,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:26,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:26,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:54:26,082 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:54:26,082 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:54:26,082 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:54:26,082 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:54:26,082 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:54:26,082 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:54:26,084 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:26,084 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:26,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:26,084 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:26,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:26,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:26,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:26,084 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:54:26,084 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:54:26,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:54:26,085 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:26,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:26,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:26,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:26,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:26,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:26,086 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:26,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:26,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:26,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:26,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:26,090 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:26,090 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:26,090 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:26,090 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:26,090 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:54:26,090 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:54:26,090 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:54:26,090 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:54:26,090 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:54:26,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:54:26,091 DEBUG [Thread-984] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:54:26,091 DEBUG [Thread-984] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:54:26,090 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:54:26,090 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:54:26,090 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:26,091 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:26,091 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:26,091 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:54:26,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:54:26,166 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(9158): Flush requested on bcfff279d7c9f90f5c57aef887110e5f 2023-04-16 00:54:26,166 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing bcfff279d7c9f90f5c57aef887110e5f 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-16 00:54:26,167 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bcfff279d7c9f90f5c57aef887110e5f, store=A 2023-04-16 00:54:26,167 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:54:26,167 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bcfff279d7c9f90f5c57aef887110e5f, store=B 2023-04-16 00:54:26,167 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:54:26,167 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bcfff279d7c9f90f5c57aef887110e5f, store=C 2023-04-16 00:54:26,167 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:54:26,191 DEBUG [Thread-984] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:54:26,191 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:54:26,193 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:54:26,194 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:54:26,194 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:54:26,194 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:54:26,194 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:54:26,194 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:54:26,194 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:54:26,195 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:26,195 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:26,196 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:26,196 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:54:26,196 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:54:26,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:54:26,196 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:26,196 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:54:26,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:26,197 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:26,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:54:26,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:26,197 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:54:26,197 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:54:26,197 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:54:26,197 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:54:26,197 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:54:26,197 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:54:26,200 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:54:26,200 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:54:26,200 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f. 2023-04-16 00:54:26,200 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f. started... 2023-04-16 00:54:26,200 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f. as already flushing 2023-04-16 00:54:26,200 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f. 2023-04-16 00:54:26,200 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:54:26,200 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:54:26,200 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:26,200 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:54:26,200 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:54:26,201 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:26,201 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:26,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:26,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:26,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:26,202 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:26,202 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:54:26,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:26,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:26,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:26,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:26,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:26,203 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:54:26,203 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:54:26,203 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@19958c90[Count = 0] remaining members to acquire global barrier 2023-04-16 00:54:26,203 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:26,204 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:26,204 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:26,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:26,204 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:26,204 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:54:26,208 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:54:26,208 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:54:26,208 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:54:26,210 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:54:26,210 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:54:26,210 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:26,210 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:54:26,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:26,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:26,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:26,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:26,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:26,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:26,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:26,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:26,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:26,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:26,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:54:26,212 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:54:26,212 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:54:26,212 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:54:26,213 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:54:26,213 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:54:26,213 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:54:26,214 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:26,214 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:26,214 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:26,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:26,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:26,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:26,214 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:54:26,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:26,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:26,215 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:26,215 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:54:26,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:26,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:54:26,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:26,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:26,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:26,216 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:26,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:26,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:26,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:26,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:26,218 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:26,218 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:26,218 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:54:26,218 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:26,218 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:54:26,218 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:54:26,218 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:54:26,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:54:26,218 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:54:26,218 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:54:26,219 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:54:26,218 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:26,219 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:26,219 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:26,219 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:54:26,219 DEBUG [Thread-984] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:54:26,219 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:26,219 DEBUG [Thread-984] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:54:26,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:54:26,235 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1681606466227, max timestamp=1681606466233Num uniques 859; 2023-04-16 00:54:26,236 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-16 00:54:26,236 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1681606466227, max timestamp=1681606466233Num uniques 859; 2023-04-16 00:54:26,236 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-16 00:54:26,236 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1681606466227, max timestamp=1681606466233Num uniques 859; 2023-04-16 00:54:26,236 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-16 00:54:26,259 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:26,259 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34680 deadline: 1681606526252, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:26,259 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:26,259 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34714 deadline: 1681606526253, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:26,259 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:26,259 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34718 deadline: 1681606526254, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:26,261 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:26,261 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34702 deadline: 1681606526258, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:26,319 DEBUG [Thread-984] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:54:26,320 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:54:26,321 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:54:26,322 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:54:26,323 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:54:26,323 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:54:26,323 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:54:26,323 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:54:26,323 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:54:26,323 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:26,323 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:26,324 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:26,324 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:54:26,325 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:54:26,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:54:26,325 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:26,325 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:54:26,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:26,325 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:26,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:54:26,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:26,326 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:54:26,326 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:54:26,326 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:54:26,326 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:54:26,326 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:54:26,326 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:54:26,326 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:54:26,326 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:54:26,326 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f. 2023-04-16 00:54:26,326 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f. started... 2023-04-16 00:54:26,326 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f. as already flushing 2023-04-16 00:54:26,326 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f. 2023-04-16 00:54:26,326 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:54:26,326 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:54:26,326 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:26,326 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:54:26,326 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:54:26,328 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:26,328 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:26,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:26,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:26,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:26,329 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:26,329 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:54:26,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:26,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:26,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:26,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:26,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:26,330 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:54:26,330 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@67d1edf7[Count = 0] remaining members to acquire global barrier 2023-04-16 00:54:26,330 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:54:26,330 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:26,332 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:26,332 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:26,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:26,332 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:54:26,332 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:54:26,332 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:54:26,332 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:26,332 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:54:26,333 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:54:26,333 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:26,333 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:54:26,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:26,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:26,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:26,334 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:54:26,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:26,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:26,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:26,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:26,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:26,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:26,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:26,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:54:26,336 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:54:26,336 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:54:26,336 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:54:26,336 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:54:26,336 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:54:26,336 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:54:26,338 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:26,338 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:26,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:26,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:26,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:26,338 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:26,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:26,338 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:54:26,338 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:54:26,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:26,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:54:26,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:26,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:26,339 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:26,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:26,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:26,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:26,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:26,341 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:26,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:26,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:26,345 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:26,345 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:26,345 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:26,345 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:54:26,345 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:54:26,345 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:54:26,345 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:54:26,345 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:26,345 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:26,345 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:26,346 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:26,345 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:54:26,345 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:54:26,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:54:26,346 DEBUG [Thread-984] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:54:26,347 DEBUG [Thread-984] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:54:26,347 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:54:26,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:54:26,362 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:26,362 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34680 deadline: 1681606526360, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:26,362 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:26,362 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34714 deadline: 1681606526360, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:26,363 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:26,363 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:26,363 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34718 deadline: 1681606526360, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:26,363 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34702 deadline: 1681606526362, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:26,447 DEBUG [Thread-984] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:54:26,448 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:54:26,450 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:54:26,451 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:54:26,451 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:54:26,451 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:54:26,451 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:54:26,451 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:54:26,451 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:54:26,451 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:26,452 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:26,453 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:54:26,453 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:26,453 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:54:26,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:54:26,453 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:26,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:26,453 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:54:26,454 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:26,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:54:26,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:26,454 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:54:26,454 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:54:26,454 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:54:26,455 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:54:26,455 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:54:26,455 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:54:26,455 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:54:26,455 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:54:26,455 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f. 2023-04-16 00:54:26,455 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f. started... 2023-04-16 00:54:26,455 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f. as already flushing 2023-04-16 00:54:26,455 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f. 2023-04-16 00:54:26,455 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:54:26,455 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:54:26,455 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:26,455 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:54:26,455 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:54:26,457 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:26,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:26,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:26,457 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:26,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:26,457 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:26,457 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:54:26,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:26,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:26,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:26,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:26,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:26,458 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:54:26,458 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4eefc1c6[Count = 0] remaining members to acquire global barrier 2023-04-16 00:54:26,458 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:54:26,458 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:26,463 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:26,463 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:26,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:26,463 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:54:26,463 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:54:26,463 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:54:26,463 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:26,463 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:54:26,464 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:26,464 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:54:26,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:26,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:26,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:26,464 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:54:26,464 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:54:26,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:26,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:26,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:26,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:26,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:26,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:26,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:26,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:54:26,466 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:54:26,466 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:54:26,467 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:54:26,467 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:54:26,467 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:54:26,467 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:54:26,468 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:26,468 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:54:26,468 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:26,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:26,468 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:26,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:26,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:26,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:26,469 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:54:26,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:54:26,469 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:26,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:26,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:26,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:26,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:26,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:26,470 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:26,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:26,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:26,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:26,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:26,472 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:26,472 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:26,472 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:26,472 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:26,472 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:54:26,472 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:54:26,472 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:54:26,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:54:26,472 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:54:26,472 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:54:26,472 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:26,473 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:54:26,473 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:54:26,473 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:54:26,473 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:26,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:54:26,473 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:26,473 DEBUG [Thread-984] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:54:26,473 DEBUG [Thread-984] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:54:26,567 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:26,567 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34714 deadline: 1681606526563, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:26,567 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:26,567 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34680 deadline: 1681606526564, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:26,567 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:26,568 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34702 deadline: 1681606526564, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:26,568 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:26,568 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34718 deadline: 1681606526566, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:26,573 DEBUG [Thread-984] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:54:26,574 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:54:26,575 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:54:26,576 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:54:26,576 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:54:26,577 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:54:26,577 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:54:26,577 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:54:26,577 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:54:26,577 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:26,577 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:26,584 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:26,584 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:54:26,584 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:54:26,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:54:26,585 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:26,585 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:54:26,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:26,585 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:26,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:54:26,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:26,586 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:54:26,586 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:54:26,586 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:54:26,586 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:54:26,587 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:54:26,587 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:54:26,587 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:54:26,587 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:54:26,587 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f. 2023-04-16 00:54:26,587 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f. started... 2023-04-16 00:54:26,587 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f. as already flushing 2023-04-16 00:54:26,587 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f. 2023-04-16 00:54:26,587 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:54:26,587 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:54:26,587 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:26,587 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:54:26,587 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:54:26,589 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:26,590 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:26,590 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:54:26,592 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:26,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:26,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:26,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:26,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:26,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:26,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:26,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:26,595 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023041676f3e7bb794e4a368c5742d28ae7215c to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041676f3e7bb794e4a368c5742d28ae7215c 2023-04-16 00:54:26,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:26,595 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:54:26,595 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5999f7c1[Count = 0] remaining members to acquire global barrier 2023-04-16 00:54:26,596 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:54:26,596 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:26,597 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:26,597 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:26,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:26,597 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:54:26,597 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:26,597 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:54:26,597 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:54:26,598 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:54:26,599 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:54:26,599 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:54:26,600 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:54:26,600 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:26,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:26,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:26,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:26,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:26,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:26,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:26,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:26,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:26,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:26,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:26,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:54:26,604 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:54:26,604 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:54:26,604 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:54:26,604 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:54:26,604 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:54:26,604 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:54:26,606 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:26,606 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:26,606 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:26,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:26,606 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:54:26,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:26,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:26,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:26,607 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:54:26,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:54:26,607 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:26,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:26,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:26,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:26,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:26,609 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:26,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:26,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:26,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:26,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:26,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:26,611 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:26,611 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:26,611 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:26,611 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:26,611 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:26,611 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:26,611 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:26,611 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:54:26,611 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:54:26,611 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:54:26,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:54:26,612 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:54:26,612 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:54:26,612 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:54:26,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:54:26,612 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:54:26,612 DEBUG [Thread-984] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:54:26,613 DEBUG [Thread-984] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:54:26,613 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:54:26,621 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=219, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/.tmp/A/26621a1cb7934592b7abc6dc6df9a986 2023-04-16 00:54:26,642 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=219 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/.tmp/B/f8ed64d1e7e040a78b5f518f4babbe79 2023-04-16 00:54:26,662 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=219 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/.tmp/C/337e0546b2704a409812eaba605aee69 2023-04-16 00:54:26,667 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/.tmp/A/26621a1cb7934592b7abc6dc6df9a986 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/A/26621a1cb7934592b7abc6dc6df9a986 2023-04-16 00:54:26,672 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/A/26621a1cb7934592b7abc6dc6df9a986, entries=300, sequenceid=219, filesize=45.8 K 2023-04-16 00:54:26,673 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/.tmp/B/f8ed64d1e7e040a78b5f518f4babbe79 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/B/f8ed64d1e7e040a78b5f518f4babbe79 2023-04-16 00:54:26,679 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/B/f8ed64d1e7e040a78b5f518f4babbe79, entries=150, sequenceid=219, filesize=11.8 K 2023-04-16 00:54:26,680 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/.tmp/C/337e0546b2704a409812eaba605aee69 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/C/337e0546b2704a409812eaba605aee69 2023-04-16 00:54:26,690 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/C/337e0546b2704a409812eaba605aee69, entries=150, sequenceid=219, filesize=11.8 K 2023-04-16 00:54:26,691 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for bcfff279d7c9f90f5c57aef887110e5f in 525ms, sequenceid=219, compaction requested=true 2023-04-16 00:54:26,691 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for bcfff279d7c9f90f5c57aef887110e5f: 2023-04-16 00:54:26,691 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 00:54:26,691 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 00:54:26,692 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-16 00:54:26,692 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-16 00:54:26,692 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 00:54:26,694 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 58026 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-16 00:54:26,694 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HStore(1912): bcfff279d7c9f90f5c57aef887110e5f/B is initiating minor compaction (all files) 2023-04-16 00:54:26,694 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HRegion(2259): Starting compaction of bcfff279d7c9f90f5c57aef887110e5f/B in TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f. 2023-04-16 00:54:26,694 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/B/8e66594f3e484d3eab1c157a1b7e145d, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/B/f02f9a14b5bb4aacaad5dc979ad36280, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/B/05ae3848c7b54920b331066dbd534638, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/B/92aca36b949c48efbb8ad2d0834d0869, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/B/f8ed64d1e7e040a78b5f518f4babbe79] into tmpdir=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/.tmp, totalSize=56.7 K 2023-04-16 00:54:26,695 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting 8e66594f3e484d3eab1c157a1b7e145d, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=134, earliestPutTs=1681606462495 2023-04-16 00:54:26,695 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 143696 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-16 00:54:26,695 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HStore(1912): bcfff279d7c9f90f5c57aef887110e5f/A is initiating minor compaction (all files) 2023-04-16 00:54:26,695 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of bcfff279d7c9f90f5c57aef887110e5f/A in TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f. 2023-04-16 00:54:26,695 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/A/ccfd106fb798442d87f12fc11d53a8e5, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/A/5d3dcad2480e49d486ddb3a48152bdf2, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/A/430045034cd042ee8842b99fce78fad7, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/A/7e589a35f7264ea482484f88a05d763c, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/A/26621a1cb7934592b7abc6dc6df9a986] into tmpdir=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/.tmp, totalSize=140.3 K 2023-04-16 00:54:26,696 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.Compactor(207): Compacting ccfd106fb798442d87f12fc11d53a8e5, keycount=150, bloomtype=ROW, size=25.5 K, encoding=NONE, compression=NONE, seqNum=134, earliestPutTs=1681606462495 2023-04-16 00:54:26,696 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting f02f9a14b5bb4aacaad5dc979ad36280, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=141, earliestPutTs=1681606462498 2023-04-16 00:54:26,696 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.Compactor(207): Compacting 5d3dcad2480e49d486ddb3a48152bdf2, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=141, earliestPutTs=1681606462498 2023-04-16 00:54:26,696 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting 05ae3848c7b54920b331066dbd534638, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=198, earliestPutTs=1681606462887 2023-04-16 00:54:26,697 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.Compactor(207): Compacting 430045034cd042ee8842b99fce78fad7, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=198, earliestPutTs=1681606462887 2023-04-16 00:54:26,697 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting 92aca36b949c48efbb8ad2d0834d0869, keycount=100, bloomtype=ROW, size=9.4 K, encoding=NONE, compression=NONE, seqNum=205, earliestPutTs=1681606464009 2023-04-16 00:54:26,697 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.Compactor(207): Compacting 7e589a35f7264ea482484f88a05d763c, keycount=100, bloomtype=ROW, size=18.5 K, encoding=NONE, compression=NONE, seqNum=205, earliestPutTs=1681606464009 2023-04-16 00:54:26,698 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting f8ed64d1e7e040a78b5f518f4babbe79, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=219, earliestPutTs=1681606466156 2023-04-16 00:54:26,698 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.Compactor(207): Compacting 26621a1cb7934592b7abc6dc6df9a986, keycount=300, bloomtype=ROW, size=45.8 K, encoding=NONE, compression=NONE, seqNum=219, earliestPutTs=1681606466145 2023-04-16 00:54:26,713 DEBUG [Thread-984] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:54:26,713 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:54:26,715 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:54:26,718 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] throttle.PressureAwareThroughputController(145): bcfff279d7c9f90f5c57aef887110e5f#B#compaction#163 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-16 00:54:26,720 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:54:26,720 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:54:26,720 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:54:26,720 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:54:26,720 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:54:26,720 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:54:26,721 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:26,721 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:26,722 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:26,722 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:54:26,722 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:54:26,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:54:26,722 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:26,722 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:54:26,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:26,723 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:26,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:54:26,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:26,723 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:54:26,725 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:54:26,725 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:54:26,725 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:54:26,725 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:54:26,725 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:54:26,727 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] throttle.PressureAwareThroughputController(145): bcfff279d7c9f90f5c57aef887110e5f#A#compaction#162 average throughput is 1.64 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-16 00:54:26,732 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:54:26,732 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:54:26,732 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f. 2023-04-16 00:54:26,732 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f. started... 2023-04-16 00:54:26,732 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing bcfff279d7c9f90f5c57aef887110e5f 3/3 column families, dataSize=127.47 KB heapSize=360.38 KB 2023-04-16 00:54:26,732 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bcfff279d7c9f90f5c57aef887110e5f, store=A 2023-04-16 00:54:26,732 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 00:54:26,732 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bcfff279d7c9f90f5c57aef887110e5f, store=B 2023-04-16 00:54:26,732 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 00:54:26,732 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bcfff279d7c9f90f5c57aef887110e5f, store=C 2023-04-16 00:54:26,733 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 00:54:26,871 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(9158): Flush requested on bcfff279d7c9f90f5c57aef887110e5f 2023-04-16 00:54:26,871 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f. as already flushing 2023-04-16 00:54:26,879 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:26,879 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34714 deadline: 1681606526877, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:26,882 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:26,882 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34702 deadline: 1681606526878, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:26,885 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:26,885 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34718 deadline: 1681606526879, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:26,885 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:26,886 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34680 deadline: 1681606526879, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:26,982 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:26,982 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34714 deadline: 1681606526980, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:26,983 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:26,984 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34702 deadline: 1681606526983, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:26,987 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:26,987 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34718 deadline: 1681606526986, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:26,988 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:26,988 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34680 deadline: 1681606526986, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:27,028 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:27,028 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34740 deadline: 1681606527026, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:27,030 DEBUG [Thread-978] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4138 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f., hostname=jenkins-hbase4.apache.org,34975,1681606419217, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor32.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-16 00:54:27,143 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/.tmp/B/0a265fceaa9f42b5a2112160556b7e93 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/B/0a265fceaa9f42b5a2112160556b7e93 2023-04-16 00:54:27,147 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in bcfff279d7c9f90f5c57aef887110e5f/B of bcfff279d7c9f90f5c57aef887110e5f into 0a265fceaa9f42b5a2112160556b7e93(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 00:54:27,147 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for bcfff279d7c9f90f5c57aef887110e5f: 2023-04-16 00:54:27,147 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f., storeName=bcfff279d7c9f90f5c57aef887110e5f/B, priority=11, startTime=1681606466691; duration=0sec 2023-04-16 00:54:27,147 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 00:54:27,148 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-16 00:54:27,149 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 58026 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-16 00:54:27,149 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HStore(1912): bcfff279d7c9f90f5c57aef887110e5f/C is initiating minor compaction (all files) 2023-04-16 00:54:27,149 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HRegion(2259): Starting compaction of bcfff279d7c9f90f5c57aef887110e5f/C in TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f. 2023-04-16 00:54:27,149 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/C/b57b50aacd47434280ba1d9b02ca28fb, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/C/d499851602f249cf8e595a65615ad78e, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/C/f537bd4395704989b5500cab8788532e, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/C/f5360c0edde043ab9f9d1b3a2ee9b3ad, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/C/337e0546b2704a409812eaba605aee69] into tmpdir=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/.tmp, totalSize=56.7 K 2023-04-16 00:54:27,150 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting b57b50aacd47434280ba1d9b02ca28fb, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=134, earliestPutTs=1681606462495 2023-04-16 00:54:27,150 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting d499851602f249cf8e595a65615ad78e, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=141, earliestPutTs=1681606462498 2023-04-16 00:54:27,150 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting f537bd4395704989b5500cab8788532e, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=198, earliestPutTs=1681606462887 2023-04-16 00:54:27,151 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting f5360c0edde043ab9f9d1b3a2ee9b3ad, keycount=100, bloomtype=ROW, size=9.4 K, encoding=NONE, compression=NONE, seqNum=205, earliestPutTs=1681606464009 2023-04-16 00:54:27,151 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting 337e0546b2704a409812eaba605aee69, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=219, earliestPutTs=1681606466156 2023-04-16 00:54:27,171 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230416528696a9e5b748f98bb9cbf10b022cd7 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230416528696a9e5b748f98bb9cbf10b022cd7 2023-04-16 00:54:27,174 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] throttle.PressureAwareThroughputController(145): bcfff279d7c9f90f5c57aef887110e5f#C#compaction#165 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-16 00:54:27,180 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=254, memsize=42.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/.tmp/A/4ccc3b61512544deb085ea689a2f6d38 2023-04-16 00:54:27,187 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:27,187 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34714 deadline: 1681606527183, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:27,187 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/.tmp/A/075edb7eb8c447e989647b62da937d3d as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/A/075edb7eb8c447e989647b62da937d3d 2023-04-16 00:54:27,187 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:27,187 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34702 deadline: 1681606527186, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:27,191 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:27,191 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34680 deadline: 1681606527189, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:27,192 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:27,193 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34718 deadline: 1681606527192, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:27,197 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in bcfff279d7c9f90f5c57aef887110e5f/A of bcfff279d7c9f90f5c57aef887110e5f into 075edb7eb8c447e989647b62da937d3d(size=25.7 K), total size for store is 25.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 00:54:27,197 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for bcfff279d7c9f90f5c57aef887110e5f: 2023-04-16 00:54:27,197 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f., storeName=bcfff279d7c9f90f5c57aef887110e5f/A, priority=11, startTime=1681606466691; duration=0sec 2023-04-16 00:54:27,198 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 00:54:27,198 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=254 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/.tmp/B/cc2a6584442c4dada7e4049ae45d3ed9 2023-04-16 00:54:27,235 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=254 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/.tmp/C/b8f3ee980a1c46929eae4e977b943d83 2023-04-16 00:54:27,240 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/.tmp/A/4ccc3b61512544deb085ea689a2f6d38 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/A/4ccc3b61512544deb085ea689a2f6d38 2023-04-16 00:54:27,245 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/A/4ccc3b61512544deb085ea689a2f6d38, entries=150, sequenceid=254, filesize=25.3 K 2023-04-16 00:54:27,246 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/.tmp/B/cc2a6584442c4dada7e4049ae45d3ed9 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/B/cc2a6584442c4dada7e4049ae45d3ed9 2023-04-16 00:54:27,250 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/B/cc2a6584442c4dada7e4049ae45d3ed9, entries=150, sequenceid=254, filesize=11.8 K 2023-04-16 00:54:27,251 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/.tmp/C/b8f3ee980a1c46929eae4e977b943d83 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/C/b8f3ee980a1c46929eae4e977b943d83 2023-04-16 00:54:27,255 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/C/b8f3ee980a1c46929eae4e977b943d83, entries=150, sequenceid=254, filesize=11.8 K 2023-04-16 00:54:27,256 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~360.33 KB/368976, currentSize=60.38 KB/61830 for bcfff279d7c9f90f5c57aef887110e5f in 524ms, sequenceid=254, compaction requested=false 2023-04-16 00:54:27,256 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for bcfff279d7c9f90f5c57aef887110e5f: 2023-04-16 00:54:27,256 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f. 2023-04-16 00:54:27,256 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:54:27,256 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:54:27,256 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:27,256 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:54:27,256 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:54:27,258 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:27,258 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:27,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:27,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:27,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:27,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:27,258 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:27,258 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:54:27,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:27,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:27,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:27,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:27,259 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:54:27,259 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6c5d016e[Count = 0] remaining members to acquire global barrier 2023-04-16 00:54:27,259 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:54:27,260 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:27,262 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:27,262 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:27,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:27,262 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:54:27,262 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:54:27,262 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:54:27,263 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:27,263 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:54:27,265 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:27,265 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:54:27,265 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:54:27,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:27,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:27,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:27,265 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:54:27,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:27,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:27,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:27,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:27,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:27,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:27,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:27,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:54:27,267 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:54:27,267 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:54:27,267 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:54:27,267 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:54:27,267 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:54:27,267 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:54:27,269 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:27,269 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:27,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:27,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:27,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:27,269 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:27,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:27,269 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:54:27,269 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:54:27,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:54:27,269 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:27,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:27,270 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:27,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:27,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:27,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:27,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:27,272 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:27,272 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:27,272 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:27,272 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:27,272 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:27,272 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:54:27,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:27,272 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:54:27,272 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:54:27,272 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:27,273 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:27,272 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:54:27,272 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:54:27,273 DEBUG [Thread-984] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:54:27,273 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:54:27,273 DEBUG [Thread-984] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:54:27,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:54:27,273 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:54:27,273 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:54:27,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:54:27,373 DEBUG [Thread-984] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:54:27,373 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:54:27,375 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:54:27,376 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:54:27,376 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:54:27,376 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:54:27,376 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:54:27,376 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:54:27,376 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:54:27,376 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:27,376 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:27,378 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:54:27,378 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:27,378 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:54:27,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:54:27,379 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:27,379 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:54:27,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:27,379 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:27,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:54:27,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:27,379 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:54:27,380 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:54:27,380 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:54:27,380 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:54:27,380 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:54:27,380 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:54:27,380 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:54:27,380 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:54:27,380 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f. 2023-04-16 00:54:27,380 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f. started... 2023-04-16 00:54:27,380 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing bcfff279d7c9f90f5c57aef887110e5f 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-16 00:54:27,380 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bcfff279d7c9f90f5c57aef887110e5f, store=A 2023-04-16 00:54:27,380 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:54:27,380 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bcfff279d7c9f90f5c57aef887110e5f, store=B 2023-04-16 00:54:27,380 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:54:27,381 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bcfff279d7c9f90f5c57aef887110e5f, store=C 2023-04-16 00:54:27,381 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:54:27,491 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(9158): Flush requested on bcfff279d7c9f90f5c57aef887110e5f 2023-04-16 00:54:27,491 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f. as already flushing 2023-04-16 00:54:27,511 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:27,511 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34680 deadline: 1681606527507, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:27,511 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:27,511 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:27,511 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34718 deadline: 1681606527507, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:27,512 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34702 deadline: 1681606527508, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:27,512 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:27,512 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34714 deadline: 1681606527509, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:27,591 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/.tmp/C/c3b7210eb7a940cea0dc85c6ec097c64 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/C/c3b7210eb7a940cea0dc85c6ec097c64 2023-04-16 00:54:27,595 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in bcfff279d7c9f90f5c57aef887110e5f/C of bcfff279d7c9f90f5c57aef887110e5f into c3b7210eb7a940cea0dc85c6ec097c64(size=12.2 K), total size for store is 23.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 00:54:27,595 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for bcfff279d7c9f90f5c57aef887110e5f: 2023-04-16 00:54:27,595 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f., storeName=bcfff279d7c9f90f5c57aef887110e5f/C, priority=11, startTime=1681606466692; duration=0sec 2023-04-16 00:54:27,595 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 00:54:27,615 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:27,615 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:27,615 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34680 deadline: 1681606527612, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:27,615 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34718 deadline: 1681606527612, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:27,615 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:27,615 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:27,615 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34702 deadline: 1681606527613, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:27,615 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34714 deadline: 1681606527613, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:27,795 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023041633808f02c44b489d8d341c40fb5b51e3 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041633808f02c44b489d8d341c40fb5b51e3 2023-04-16 00:54:27,802 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=268, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/.tmp/A/fe13155d833d4c6984e41d4f025dcc10 2023-04-16 00:54:27,815 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=268 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/.tmp/B/d6ae0991afa24736b5c709a5a60beb00 2023-04-16 00:54:27,819 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:27,820 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34718 deadline: 1681606527817, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:27,820 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:27,820 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:27,820 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34702 deadline: 1681606527817, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:27,820 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34680 deadline: 1681606527818, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:27,820 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:27,820 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34714 deadline: 1681606527818, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:27,831 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=268 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/.tmp/C/8a308f513dd94800b86ab205377d09a9 2023-04-16 00:54:27,836 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/.tmp/A/fe13155d833d4c6984e41d4f025dcc10 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/A/fe13155d833d4c6984e41d4f025dcc10 2023-04-16 00:54:27,841 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/A/fe13155d833d4c6984e41d4f025dcc10, entries=150, sequenceid=268, filesize=25.4 K 2023-04-16 00:54:27,842 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/.tmp/B/d6ae0991afa24736b5c709a5a60beb00 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/B/d6ae0991afa24736b5c709a5a60beb00 2023-04-16 00:54:27,847 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/B/d6ae0991afa24736b5c709a5a60beb00, entries=150, sequenceid=268, filesize=11.9 K 2023-04-16 00:54:27,850 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/.tmp/C/8a308f513dd94800b86ab205377d09a9 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/C/8a308f513dd94800b86ab205377d09a9 2023-04-16 00:54:27,854 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/C/8a308f513dd94800b86ab205377d09a9, entries=150, sequenceid=268, filesize=11.9 K 2023-04-16 00:54:27,855 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for bcfff279d7c9f90f5c57aef887110e5f in 475ms, sequenceid=268, compaction requested=true 2023-04-16 00:54:27,855 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for bcfff279d7c9f90f5c57aef887110e5f: 2023-04-16 00:54:27,855 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f. 2023-04-16 00:54:27,855 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:54:27,855 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:54:27,855 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:27,855 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:54:27,855 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:54:27,857 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:27,857 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:27,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:27,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:27,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:27,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:27,857 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:27,858 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:54:27,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:27,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:27,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:27,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:27,859 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:54:27,859 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@40be747[Count = 0] remaining members to acquire global barrier 2023-04-16 00:54:27,859 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:54:27,859 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:27,860 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:27,860 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:27,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:27,860 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:54:27,860 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:54:27,860 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:54:27,866 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:54:27,866 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:54:27,866 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:54:27,866 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:27,867 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:54:27,867 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:54:27,867 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:54:27,867 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:54:27,867 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:54:27,867 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:54:27,867 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:54:27,869 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:27,869 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:27,869 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:54:27,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:27,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:27,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:27,869 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:27,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:27,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:27,870 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:54:27,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:54:27,870 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:27,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:27,871 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:27,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:27,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:27,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:27,874 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:27,874 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:27,874 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:27,874 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:27,874 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:54:27,874 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:54:27,875 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:54:27,874 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:27,875 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:27,875 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:27,875 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:54:27,875 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:54:27,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:27,875 DEBUG [Thread-984] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:54:27,875 DEBUG [Thread-984] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:54:27,875 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:54:27,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:54:27,876 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:54:27,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:54:27,878 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:54:27,975 DEBUG [Thread-984] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:54:27,976 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:54:27,978 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:54:27,979 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:54:27,979 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:54:27,979 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:54:27,979 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:54:27,979 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:54:27,979 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:54:27,979 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:27,979 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:27,980 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:54:27,980 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:54:27,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:54:27,980 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:27,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:27,981 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:27,981 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:54:27,981 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:27,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:54:27,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:27,982 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:54:27,982 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:54:27,982 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:54:27,982 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:54:27,982 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:54:27,982 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:54:27,982 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:54:27,982 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:54:27,982 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f. 2023-04-16 00:54:27,982 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f. started... 2023-04-16 00:54:27,982 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing bcfff279d7c9f90f5c57aef887110e5f 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-16 00:54:27,982 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bcfff279d7c9f90f5c57aef887110e5f, store=A 2023-04-16 00:54:27,983 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:54:27,983 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bcfff279d7c9f90f5c57aef887110e5f, store=B 2023-04-16 00:54:27,983 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:54:27,983 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bcfff279d7c9f90f5c57aef887110e5f, store=C 2023-04-16 00:54:27,983 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:54:28,001 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230416a81b747b432544c181065018b419dd26 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230416a81b747b432544c181065018b419dd26 2023-04-16 00:54:28,010 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=289, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/.tmp/A/a6893ea908c745e097b7ab68ba2b400a 2023-04-16 00:54:28,028 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=289 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/.tmp/B/a2f611f82f4b48f0b1f66b1d3e98763f 2023-04-16 00:54:28,041 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=289 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/.tmp/C/0576db228c8842cc83f5d67c893bc1c4 2023-04-16 00:54:28,047 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/.tmp/A/a6893ea908c745e097b7ab68ba2b400a as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/A/a6893ea908c745e097b7ab68ba2b400a 2023-04-16 00:54:28,051 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/A/a6893ea908c745e097b7ab68ba2b400a, entries=150, sequenceid=289, filesize=25.4 K 2023-04-16 00:54:28,052 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/.tmp/B/a2f611f82f4b48f0b1f66b1d3e98763f as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/B/a2f611f82f4b48f0b1f66b1d3e98763f 2023-04-16 00:54:28,057 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/B/a2f611f82f4b48f0b1f66b1d3e98763f, entries=150, sequenceid=289, filesize=11.9 K 2023-04-16 00:54:28,059 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/.tmp/C/0576db228c8842cc83f5d67c893bc1c4 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/C/0576db228c8842cc83f5d67c893bc1c4 2023-04-16 00:54:28,062 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/C/0576db228c8842cc83f5d67c893bc1c4, entries=150, sequenceid=289, filesize=11.9 K 2023-04-16 00:54:28,063 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=0 B/0 for bcfff279d7c9f90f5c57aef887110e5f in 81ms, sequenceid=289, compaction requested=true 2023-04-16 00:54:28,063 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for bcfff279d7c9f90f5c57aef887110e5f: 2023-04-16 00:54:28,063 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f. 2023-04-16 00:54:28,064 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:54:28,064 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:54:28,064 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:28,064 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:54:28,064 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:54:28,067 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:28,067 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:28,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:28,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:28,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:28,067 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:28,067 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:54:28,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:28,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:28,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:28,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:28,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:28,070 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:54:28,070 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@34dae7f1[Count = 0] remaining members to acquire global barrier 2023-04-16 00:54:28,070 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:54:28,070 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:28,071 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:28,071 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:28,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:28,072 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:54:28,072 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:54:28,072 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:54:28,072 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:28,072 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:54:28,073 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:28,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:28,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:28,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:28,073 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:54:28,074 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:54:28,074 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:54:28,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:28,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:28,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:28,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:28,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:28,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:28,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:28,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:54:28,077 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:54:28,077 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:54:28,077 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:54:28,077 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:54:28,077 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:54:28,077 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:54:28,078 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:28,078 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:28,078 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:28,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:28,078 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:54:28,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:28,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:28,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:28,079 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:54:28,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:54:28,079 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:28,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:28,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:28,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:28,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:28,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:28,081 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:28,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:28,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:28,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:28,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:28,085 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:28,085 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:54:28,085 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:28,085 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:54:28,085 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:54:28,085 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:28,085 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:28,085 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:28,085 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:28,085 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:28,085 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:54:28,085 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:54:28,085 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:54:28,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:54:28,085 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:54:28,085 DEBUG [Thread-984] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:54:28,085 DEBUG [Thread-984] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:54:28,086 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:54:28,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:54:28,151 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing bcfff279d7c9f90f5c57aef887110e5f 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-16 00:54:28,152 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bcfff279d7c9f90f5c57aef887110e5f, store=A 2023-04-16 00:54:28,152 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:54:28,152 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bcfff279d7c9f90f5c57aef887110e5f, store=B 2023-04-16 00:54:28,152 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(9158): Flush requested on bcfff279d7c9f90f5c57aef887110e5f 2023-04-16 00:54:28,152 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:54:28,152 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bcfff279d7c9f90f5c57aef887110e5f, store=C 2023-04-16 00:54:28,152 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:54:28,186 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1681606468173, max timestamp=1681606468184Num uniques 859; 2023-04-16 00:54:28,187 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-16 00:54:28,187 DEBUG [Thread-984] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:54:28,188 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1681606468173, max timestamp=1681606468184Num uniques 859; 2023-04-16 00:54:28,189 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-16 00:54:28,189 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1681606468173, max timestamp=1681606468184Num uniques 859; 2023-04-16 00:54:28,189 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-16 00:54:28,190 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:54:28,193 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:54:28,193 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230416d67a5ffb9cb148e1ab9ada4c26e28412 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230416d67a5ffb9cb148e1ab9ada4c26e28412 2023-04-16 00:54:28,194 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:54:28,194 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:54:28,194 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:54:28,194 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:54:28,194 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:54:28,194 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:54:28,195 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:28,195 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:28,197 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:28,197 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:54:28,197 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:54:28,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:54:28,197 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:28,197 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:54:28,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:28,197 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:28,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:54:28,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:28,198 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:54:28,202 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:54:28,202 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:54:28,202 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:54:28,202 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:54:28,202 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:54:28,212 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:54:28,212 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:54:28,212 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f. 2023-04-16 00:54:28,212 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f. started... 2023-04-16 00:54:28,212 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f. as already flushing 2023-04-16 00:54:28,212 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f. 2023-04-16 00:54:28,212 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:54:28,212 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:54:28,212 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:28,213 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:54:28,213 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:54:28,214 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:28,214 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:28,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:28,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:28,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:28,215 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:28,215 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:54:28,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:28,215 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:28,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:28,215 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34714 deadline: 1681606528208, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:28,216 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:28,216 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34718 deadline: 1681606528209, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:28,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:28,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:28,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:28,217 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:54:28,217 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@226b2954[Count = 0] remaining members to acquire global barrier 2023-04-16 00:54:28,217 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:54:28,217 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:28,218 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:28,218 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:28,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:28,218 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:54:28,218 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:54:28,218 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:54:28,218 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:28,218 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:54:28,218 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:28,219 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34680 deadline: 1681606528210, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:28,220 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:28,221 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34702 deadline: 1681606528215, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:28,222 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=299, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/.tmp/A/d4a18a42bce44d68a6de859d66c68a1c 2023-04-16 00:54:28,226 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:54:28,226 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:54:28,226 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:54:28,226 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:28,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:28,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:28,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:28,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:28,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:28,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:28,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:28,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:28,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:28,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:28,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:54:28,231 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:54:28,231 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:54:28,231 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:54:28,231 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:54:28,231 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:54:28,231 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:54:28,234 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:28,234 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:54:28,234 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:28,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:28,234 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:28,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:28,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:28,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:28,234 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:54:28,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:54:28,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:28,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:28,235 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:28,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:28,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:28,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:28,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:28,236 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:28,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:28,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:28,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:28,239 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:54:28,239 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:54:28,239 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:54:28,239 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:28,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:54:28,239 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:54:28,239 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:54:28,239 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:54:28,239 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:28,239 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:28,239 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:28,239 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:28,240 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:28,240 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:28,240 DEBUG [Thread-984] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:54:28,240 DEBUG [Thread-984] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:54:28,240 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:54:28,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:54:28,258 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=299 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/.tmp/B/c82b06a6aae44ad8a242e5105d206819 2023-04-16 00:54:28,321 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:28,321 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:28,321 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34714 deadline: 1681606528317, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:28,321 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34718 deadline: 1681606528317, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:28,321 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:28,321 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34680 deadline: 1681606528319, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:28,322 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:28,322 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34702 deadline: 1681606528322, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:28,340 DEBUG [Thread-984] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:54:28,340 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:54:28,342 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:54:28,343 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:54:28,343 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:54:28,343 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:54:28,343 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:54:28,343 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:54:28,343 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:54:28,343 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:28,343 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:28,344 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:28,344 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:54:28,345 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:54:28,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:54:28,345 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:28,345 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:54:28,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:28,345 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:28,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:54:28,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:28,346 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:54:28,346 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:54:28,346 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:54:28,346 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:54:28,346 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:54:28,346 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:54:28,346 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:54:28,346 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:54:28,346 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f. 2023-04-16 00:54:28,347 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f. started... 2023-04-16 00:54:28,347 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f. as already flushing 2023-04-16 00:54:28,347 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f. 2023-04-16 00:54:28,347 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:54:28,347 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:54:28,347 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:28,347 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:54:28,347 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:54:28,348 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:28,348 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:28,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:28,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:28,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:28,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:28,349 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:28,349 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:54:28,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:28,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:28,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:28,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:28,350 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:54:28,350 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:54:28,350 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@26b90a06[Count = 0] remaining members to acquire global barrier 2023-04-16 00:54:28,350 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:28,351 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:28,351 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:28,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:28,352 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:54:28,352 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:54:28,352 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:28,352 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:54:28,352 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:54:28,358 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:54:28,358 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:54:28,358 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:28,358 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:54:28,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:28,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:28,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:28,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:28,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:28,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:28,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:28,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:28,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:28,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:28,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:54:28,361 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:54:28,361 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:54:28,361 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:54:28,361 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:54:28,361 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:54:28,361 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:54:28,362 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:28,362 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:28,362 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:54:28,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:28,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:28,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:28,362 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:28,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:28,363 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:54:28,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:54:28,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:28,363 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:28,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:28,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:28,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:28,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:28,365 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:28,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:28,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:28,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:28,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:28,370 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:54:28,370 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:28,370 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:54:28,370 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:54:28,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:54:28,370 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:54:28,370 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:28,370 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:54:28,370 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:54:28,370 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:28,371 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:28,371 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:54:28,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:54:28,371 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:28,371 DEBUG [Thread-984] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:54:28,371 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:28,371 DEBUG [Thread-984] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:54:28,371 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:28,471 DEBUG [Thread-984] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:54:28,472 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:54:28,474 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:54:28,475 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:54:28,476 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:54:28,476 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:54:28,476 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:54:28,476 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:54:28,476 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:54:28,476 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:28,476 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:28,477 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:54:28,477 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:54:28,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:54:28,478 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:28,478 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:28,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:28,478 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:54:28,478 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:28,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:54:28,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:28,479 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:54:28,479 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:54:28,479 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:54:28,479 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:54:28,479 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:54:28,479 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:54:28,479 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:54:28,479 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:54:28,479 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f. 2023-04-16 00:54:28,479 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f. started... 2023-04-16 00:54:28,480 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f. as already flushing 2023-04-16 00:54:28,480 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f. 2023-04-16 00:54:28,480 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:54:28,480 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:54:28,480 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:28,480 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:54:28,480 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:54:28,481 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:28,481 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:28,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:28,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:28,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:28,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:28,482 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:28,482 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:54:28,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:28,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:28,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:28,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:28,483 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:54:28,483 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:54:28,483 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7d53f049[Count = 0] remaining members to acquire global barrier 2023-04-16 00:54:28,483 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:28,485 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:28,485 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:28,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:28,485 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:54:28,485 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:54:28,485 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:54:28,485 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:28,485 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:54:28,486 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:54:28,486 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:54:28,486 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:54:28,486 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:28,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:28,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:28,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:28,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:28,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:28,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:28,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:28,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:28,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:28,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:28,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:54:28,496 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:54:28,496 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:54:28,496 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:54:28,496 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:54:28,496 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:54:28,496 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:54:28,498 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:28,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:28,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:28,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:28,498 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:28,498 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:54:28,498 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:28,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:28,499 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:54:28,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:54:28,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:28,499 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:28,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:28,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:28,500 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:28,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:28,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:28,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:28,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:28,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:28,503 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:28,503 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:28,503 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:28,503 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:54:28,503 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:54:28,503 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:54:28,503 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:54:28,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:54:28,503 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:54:28,503 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:54:28,503 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:28,504 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:28,504 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:28,504 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:28,503 DEBUG [Thread-984] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:54:28,504 DEBUG [Thread-984] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:54:28,504 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:54:28,504 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:54:28,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:54:28,524 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:28,524 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34718 deadline: 1681606528522, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:28,524 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:28,524 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34714 deadline: 1681606528522, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:28,524 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:28,524 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34680 deadline: 1681606528523, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:28,524 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:28,524 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34702 deadline: 1681606528524, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:28,604 DEBUG [Thread-984] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:54:28,605 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:54:28,606 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:54:28,607 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:54:28,607 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:54:28,607 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:54:28,607 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:54:28,607 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:54:28,607 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:54:28,608 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:28,608 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:28,610 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:54:28,610 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:28,610 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:54:28,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:54:28,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:28,610 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:28,610 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:54:28,610 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:28,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:54:28,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:28,611 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:54:28,611 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:54:28,611 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:54:28,611 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:54:28,611 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:54:28,611 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:54:28,611 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:54:28,611 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:54:28,612 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f. 2023-04-16 00:54:28,612 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f. started... 2023-04-16 00:54:28,612 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f. as already flushing 2023-04-16 00:54:28,612 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f. 2023-04-16 00:54:28,612 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:54:28,612 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:54:28,612 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:28,612 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:54:28,612 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:54:28,613 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:28,613 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:28,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:28,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:28,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:28,613 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:28,613 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:54:28,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:28,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:28,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:28,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:28,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:28,615 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:54:28,615 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:54:28,615 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1ed5757d[Count = 0] remaining members to acquire global barrier 2023-04-16 00:54:28,615 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:28,616 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:28,616 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:28,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:28,616 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:54:28,616 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:54:28,616 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:54:28,616 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:28,616 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:54:28,617 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:28,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:28,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:28,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:28,617 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:54:28,618 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:54:28,618 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:54:28,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:28,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:28,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:28,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:28,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:28,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:28,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:28,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:54:28,627 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:54:28,627 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:54:28,627 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:54:28,627 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:54:28,627 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:54:28,627 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:54:28,630 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:28,630 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:28,630 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:28,630 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:54:28,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:28,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:28,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:28,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:28,630 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:54:28,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:54:28,631 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:28,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:28,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:28,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:28,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:28,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:28,632 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:28,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:28,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:28,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:28,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:28,634 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:54:28,634 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:28,634 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:54:28,634 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:54:28,634 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:54:28,634 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:54:28,634 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:54:28,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:54:28,634 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:28,634 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:28,634 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:28,634 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:28,634 DEBUG [Thread-984] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:54:28,634 DEBUG [Thread-984] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:54:28,634 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:28,635 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:28,635 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:54:28,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:54:28,674 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=299 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/.tmp/C/c728232926ba42d0bc05b59493372d69 2023-04-16 00:54:28,679 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/.tmp/A/d4a18a42bce44d68a6de859d66c68a1c as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/A/d4a18a42bce44d68a6de859d66c68a1c 2023-04-16 00:54:28,682 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/A/d4a18a42bce44d68a6de859d66c68a1c, entries=350, sequenceid=299, filesize=52.9 K 2023-04-16 00:54:28,683 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/.tmp/B/c82b06a6aae44ad8a242e5105d206819 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/B/c82b06a6aae44ad8a242e5105d206819 2023-04-16 00:54:28,687 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/B/c82b06a6aae44ad8a242e5105d206819, entries=150, sequenceid=299, filesize=11.9 K 2023-04-16 00:54:28,687 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/.tmp/C/c728232926ba42d0bc05b59493372d69 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/C/c728232926ba42d0bc05b59493372d69 2023-04-16 00:54:28,691 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/C/c728232926ba42d0bc05b59493372d69, entries=150, sequenceid=299, filesize=11.9 K 2023-04-16 00:54:28,692 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=140.89 KB/144270 for bcfff279d7c9f90f5c57aef887110e5f in 541ms, sequenceid=299, compaction requested=true 2023-04-16 00:54:28,692 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for bcfff279d7c9f90f5c57aef887110e5f: 2023-04-16 00:54:28,692 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 00:54:28,692 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-16 00:54:28,692 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-16 00:54:28,692 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 00:54:28,692 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 00:54:28,694 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 158508 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-16 00:54:28,694 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HStore(1912): bcfff279d7c9f90f5c57aef887110e5f/A is initiating minor compaction (all files) 2023-04-16 00:54:28,694 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61040 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-16 00:54:28,694 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of bcfff279d7c9f90f5c57aef887110e5f/A in TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f. 2023-04-16 00:54:28,694 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HStore(1912): bcfff279d7c9f90f5c57aef887110e5f/B is initiating minor compaction (all files) 2023-04-16 00:54:28,694 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/A/075edb7eb8c447e989647b62da937d3d, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/A/4ccc3b61512544deb085ea689a2f6d38, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/A/fe13155d833d4c6984e41d4f025dcc10, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/A/a6893ea908c745e097b7ab68ba2b400a, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/A/d4a18a42bce44d68a6de859d66c68a1c] into tmpdir=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/.tmp, totalSize=154.8 K 2023-04-16 00:54:28,694 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HRegion(2259): Starting compaction of bcfff279d7c9f90f5c57aef887110e5f/B in TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f. 2023-04-16 00:54:28,694 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/B/0a265fceaa9f42b5a2112160556b7e93, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/B/cc2a6584442c4dada7e4049ae45d3ed9, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/B/d6ae0991afa24736b5c709a5a60beb00, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/B/a2f611f82f4b48f0b1f66b1d3e98763f, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/B/c82b06a6aae44ad8a242e5105d206819] into tmpdir=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/.tmp, totalSize=59.6 K 2023-04-16 00:54:28,694 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.Compactor(207): Compacting 075edb7eb8c447e989647b62da937d3d, keycount=150, bloomtype=ROW, size=25.7 K, encoding=NONE, compression=NONE, seqNum=219, earliestPutTs=1681606466156 2023-04-16 00:54:28,694 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting 0a265fceaa9f42b5a2112160556b7e93, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=219, earliestPutTs=1681606466156 2023-04-16 00:54:28,695 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.Compactor(207): Compacting 4ccc3b61512544deb085ea689a2f6d38, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=254, earliestPutTs=1681606466250 2023-04-16 00:54:28,696 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting cc2a6584442c4dada7e4049ae45d3ed9, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=254, earliestPutTs=1681606466250 2023-04-16 00:54:28,696 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.Compactor(207): Compacting fe13155d833d4c6984e41d4f025dcc10, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=268, earliestPutTs=1681606466874 2023-04-16 00:54:28,697 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting d6ae0991afa24736b5c709a5a60beb00, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=268, earliestPutTs=1681606466874 2023-04-16 00:54:28,697 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.Compactor(207): Compacting a6893ea908c745e097b7ab68ba2b400a, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=289, earliestPutTs=1681606467505 2023-04-16 00:54:28,697 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting a2f611f82f4b48f0b1f66b1d3e98763f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=289, earliestPutTs=1681606467505 2023-04-16 00:54:28,697 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.Compactor(207): Compacting d4a18a42bce44d68a6de859d66c68a1c, keycount=350, bloomtype=ROW, size=52.9 K, encoding=NONE, compression=NONE, seqNum=299, earliestPutTs=1681606468143 2023-04-16 00:54:28,697 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting c82b06a6aae44ad8a242e5105d206819, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=299, earliestPutTs=1681606468148 2023-04-16 00:54:28,709 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] throttle.PressureAwareThroughputController(145): bcfff279d7c9f90f5c57aef887110e5f#B#compaction#178 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-16 00:54:28,711 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] throttle.PressureAwareThroughputController(145): bcfff279d7c9f90f5c57aef887110e5f#A#compaction#177 average throughput is 3.94 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-16 00:54:28,724 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/.tmp/B/21b806fc0f7f41c2b61745a3181c339e as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/B/21b806fc0f7f41c2b61745a3181c339e 2023-04-16 00:54:28,728 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in bcfff279d7c9f90f5c57aef887110e5f/B of bcfff279d7c9f90f5c57aef887110e5f into 21b806fc0f7f41c2b61745a3181c339e(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 00:54:28,728 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for bcfff279d7c9f90f5c57aef887110e5f: 2023-04-16 00:54:28,728 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f., storeName=bcfff279d7c9f90f5c57aef887110e5f/B, priority=11, startTime=1681606468692; duration=0sec 2023-04-16 00:54:28,729 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 00:54:28,729 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-16 00:54:28,730 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61040 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-16 00:54:28,730 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HStore(1912): bcfff279d7c9f90f5c57aef887110e5f/C is initiating minor compaction (all files) 2023-04-16 00:54:28,730 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HRegion(2259): Starting compaction of bcfff279d7c9f90f5c57aef887110e5f/C in TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f. 2023-04-16 00:54:28,730 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/C/c3b7210eb7a940cea0dc85c6ec097c64, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/C/b8f3ee980a1c46929eae4e977b943d83, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/C/8a308f513dd94800b86ab205377d09a9, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/C/0576db228c8842cc83f5d67c893bc1c4, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/C/c728232926ba42d0bc05b59493372d69] into tmpdir=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/.tmp, totalSize=59.6 K 2023-04-16 00:54:28,731 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting c3b7210eb7a940cea0dc85c6ec097c64, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=219, earliestPutTs=1681606466156 2023-04-16 00:54:28,732 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting b8f3ee980a1c46929eae4e977b943d83, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=254, earliestPutTs=1681606466250 2023-04-16 00:54:28,732 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting 8a308f513dd94800b86ab205377d09a9, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=268, earliestPutTs=1681606466874 2023-04-16 00:54:28,732 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting 0576db228c8842cc83f5d67c893bc1c4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=289, earliestPutTs=1681606467505 2023-04-16 00:54:28,733 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting c728232926ba42d0bc05b59493372d69, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=299, earliestPutTs=1681606468148 2023-04-16 00:54:28,735 DEBUG [Thread-984] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:54:28,735 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:54:28,737 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:54:28,738 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:54:28,739 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:54:28,739 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:54:28,739 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:54:28,739 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:54:28,739 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:54:28,739 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:28,739 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:28,740 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:28,740 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:54:28,740 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:54:28,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:54:28,741 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:28,741 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:54:28,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:28,741 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:28,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:54:28,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:28,742 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:54:28,742 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] throttle.PressureAwareThroughputController(145): bcfff279d7c9f90f5c57aef887110e5f#C#compaction#179 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-16 00:54:28,742 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:54:28,743 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:54:28,743 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:54:28,743 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:54:28,743 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:54:28,743 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:54:28,743 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:54:28,743 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f. 2023-04-16 00:54:28,743 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f. started... 2023-04-16 00:54:28,743 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing bcfff279d7c9f90f5c57aef887110e5f 3/3 column families, dataSize=140.89 KB heapSize=390.84 KB 2023-04-16 00:54:28,744 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bcfff279d7c9f90f5c57aef887110e5f, store=A 2023-04-16 00:54:28,744 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 00:54:28,744 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bcfff279d7c9f90f5c57aef887110e5f, store=B 2023-04-16 00:54:28,744 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 00:54:28,744 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bcfff279d7c9f90f5c57aef887110e5f, store=C 2023-04-16 00:54:28,744 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 00:54:28,762 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230416f102ed9292df49528bff1ce49ec69238 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230416f102ed9292df49528bff1ce49ec69238 2023-04-16 00:54:28,826 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(9158): Flush requested on bcfff279d7c9f90f5c57aef887110e5f 2023-04-16 00:54:28,827 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f. as already flushing 2023-04-16 00:54:28,837 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:28,837 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34702 deadline: 1681606528834, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:28,837 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:28,837 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34680 deadline: 1681606528835, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:28,837 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:28,837 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34714 deadline: 1681606528836, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:28,837 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:28,838 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34718 deadline: 1681606528837, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:28,940 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:28,940 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34680 deadline: 1681606528938, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:28,940 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:28,940 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34702 deadline: 1681606528938, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:28,940 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:28,940 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34714 deadline: 1681606528938, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:28,940 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:28,941 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34718 deadline: 1681606528938, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:29,142 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:29,142 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34702 deadline: 1681606529141, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:29,142 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:29,142 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34718 deadline: 1681606529141, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:29,142 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:29,143 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34680 deadline: 1681606529142, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:29,143 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:29,143 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34714 deadline: 1681606529142, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:29,157 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/.tmp/C/54170cd1ceda4274a259fe41ee4da8f8 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/C/54170cd1ceda4274a259fe41ee4da8f8 2023-04-16 00:54:29,162 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in bcfff279d7c9f90f5c57aef887110e5f/C of bcfff279d7c9f90f5c57aef887110e5f into 54170cd1ceda4274a259fe41ee4da8f8(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 00:54:29,162 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for bcfff279d7c9f90f5c57aef887110e5f: 2023-04-16 00:54:29,162 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f., storeName=bcfff279d7c9f90f5c57aef887110e5f/C, priority=11, startTime=1681606468692; duration=0sec 2023-04-16 00:54:29,162 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 00:54:29,169 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=335, memsize=47.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/.tmp/A/294c519045464f2ebd5223681bbdaa69 2023-04-16 00:54:29,444 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:29,445 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34714 deadline: 1681606529444, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:29,445 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:29,445 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34718 deadline: 1681606529444, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:29,447 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:29,448 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34702 deadline: 1681606529445, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:29,448 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:29,448 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34680 deadline: 1681606529445, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:29,582 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=335 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/.tmp/B/640431c713c94b3bb44acca27ba0b0ca 2023-04-16 00:54:29,938 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/.tmp/A/c274a82a5dfb44d3a9d5d0f8b55f9678 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/A/c274a82a5dfb44d3a9d5d0f8b55f9678 2023-04-16 00:54:29,942 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in bcfff279d7c9f90f5c57aef887110e5f/A of bcfff279d7c9f90f5c57aef887110e5f into c274a82a5dfb44d3a9d5d0f8b55f9678(size=26.0 K), total size for store is 26.0 K. This selection was in queue for 0sec, and took 1sec to execute. 2023-04-16 00:54:29,942 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for bcfff279d7c9f90f5c57aef887110e5f: 2023-04-16 00:54:29,942 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f., storeName=bcfff279d7c9f90f5c57aef887110e5f/A, priority=11, startTime=1681606468692; duration=1sec 2023-04-16 00:54:29,943 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 00:54:29,947 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:29,947 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34718 deadline: 1681606529946, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:29,949 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:29,950 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34714 deadline: 1681606529948, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:29,952 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:29,953 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34702 deadline: 1681606529951, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:29,953 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:29,953 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34680 deadline: 1681606529951, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:29,992 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=335 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/.tmp/C/621791b9c43346cdbf24e4d25984a701 2023-04-16 00:54:29,997 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/.tmp/A/294c519045464f2ebd5223681bbdaa69 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/A/294c519045464f2ebd5223681bbdaa69 2023-04-16 00:54:30,001 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/A/294c519045464f2ebd5223681bbdaa69, entries=150, sequenceid=335, filesize=25.4 K 2023-04-16 00:54:30,001 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/.tmp/B/640431c713c94b3bb44acca27ba0b0ca as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/B/640431c713c94b3bb44acca27ba0b0ca 2023-04-16 00:54:30,008 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/B/640431c713c94b3bb44acca27ba0b0ca, entries=150, sequenceid=335, filesize=11.9 K 2023-04-16 00:54:30,009 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/.tmp/C/621791b9c43346cdbf24e4d25984a701 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/C/621791b9c43346cdbf24e4d25984a701 2023-04-16 00:54:30,012 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/C/621791b9c43346cdbf24e4d25984a701, entries=150, sequenceid=335, filesize=11.9 K 2023-04-16 00:54:30,013 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~390.80 KB/400176, currentSize=53.67 KB/54960 for bcfff279d7c9f90f5c57aef887110e5f in 1270ms, sequenceid=335, compaction requested=false 2023-04-16 00:54:30,013 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for bcfff279d7c9f90f5c57aef887110e5f: 2023-04-16 00:54:30,013 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f. 2023-04-16 00:54:30,013 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:54:30,013 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:54:30,013 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:30,013 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:54:30,014 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:54:30,015 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:30,015 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:30,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:30,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:30,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:30,015 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:30,016 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:54:30,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:30,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:30,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:30,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:30,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:30,017 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:54:30,017 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@658fde7a[Count = 0] remaining members to acquire global barrier 2023-04-16 00:54:30,017 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:54:30,018 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:30,019 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:30,020 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:30,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:30,020 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:54:30,020 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:54:30,020 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:54:30,020 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:30,020 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:54:30,022 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:30,022 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:54:30,022 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:54:30,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:30,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:30,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:30,022 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:54:30,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:30,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:30,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:30,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:30,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:30,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:30,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:30,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:54:30,025 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:54:30,025 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:54:30,025 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:54:30,025 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:54:30,025 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:54:30,025 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:54:30,027 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:30,027 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:30,027 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:30,027 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:54:30,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:30,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:30,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:30,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:30,028 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:54:30,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:54:30,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:30,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:30,029 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:30,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:30,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:30,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:30,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:30,030 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:30,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:30,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:30,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:30,037 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:30,037 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:30,037 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:30,037 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:30,037 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:54:30,037 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:54:30,037 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:54:30,037 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:54:30,037 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:54:30,037 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:30,037 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:30,037 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:30,037 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:54:30,037 DEBUG [Thread-984] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:54:30,037 DEBUG [Thread-984] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:54:30,037 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:54:30,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:54:30,038 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:54:30,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:54:30,137 DEBUG [Thread-984] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:54:30,138 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:54:30,139 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:54:30,140 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:54:30,141 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:54:30,141 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:54:30,141 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:54:30,141 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:54:30,141 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:54:30,141 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:30,141 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:30,143 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:54:30,143 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:30,143 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:54:30,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:54:30,143 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:30,143 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:54:30,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:30,144 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:30,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:54:30,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:30,144 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:54:30,144 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:54:30,144 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:54:30,144 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:54:30,145 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:54:30,145 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:54:30,145 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:54:30,145 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:54:30,145 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f. 2023-04-16 00:54:30,145 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f. started... 2023-04-16 00:54:30,145 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing bcfff279d7c9f90f5c57aef887110e5f 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-16 00:54:30,145 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bcfff279d7c9f90f5c57aef887110e5f, store=A 2023-04-16 00:54:30,145 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:54:30,145 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bcfff279d7c9f90f5c57aef887110e5f, store=B 2023-04-16 00:54:30,145 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:54:30,145 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bcfff279d7c9f90f5c57aef887110e5f, store=C 2023-04-16 00:54:30,145 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:54:30,563 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304168becbef46c2d43dd896d4c3646710d70 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304168becbef46c2d43dd896d4c3646710d70 2023-04-16 00:54:30,952 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(9158): Flush requested on bcfff279d7c9f90f5c57aef887110e5f 2023-04-16 00:54:30,952 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f. as already flushing 2023-04-16 00:54:30,970 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=348, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/.tmp/A/0be1677bce94454eb2808230b3f443de 2023-04-16 00:54:30,975 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=409, cellSize=18725, totalHeapSize=42776, min timestamp=1681606470960, max timestamp=1681606470974Num uniques 859; 2023-04-16 00:54:30,975 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=18725, suffix heap size=117064,new segment heap size=42776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=409 2023-04-16 00:54:30,975 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=409, cellSize=18725, totalHeapSize=42776, min timestamp=1681606470960, max timestamp=1681606470974Num uniques 859; 2023-04-16 00:54:30,976 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=409, cellSize=18725, totalHeapSize=42776, min timestamp=1681606470960, max timestamp=1681606470974Num uniques 859; 2023-04-16 00:54:30,976 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=18725, suffix heap size=117064,new segment heap size=42776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=409 2023-04-16 00:54:30,976 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=18725, suffix heap size=117064,new segment heap size=42776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=409 2023-04-16 00:54:30,981 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=348 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/.tmp/B/c7e598dcb45e40aea0c5899f5dd1c491 2023-04-16 00:54:30,991 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:30,992 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34718 deadline: 1681606530987, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:30,992 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:30,992 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:30,992 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 159 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34702 deadline: 1681606530989, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:30,992 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34680 deadline: 1681606530988, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:30,992 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:30,992 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34714 deadline: 1681606530989, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:31,063 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:31,063 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34740 deadline: 1681606531062, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:31,064 DEBUG [Thread-978] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8172 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f., hostname=jenkins-hbase4.apache.org,34975,1681606419217, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor32.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-16 00:54:31,094 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:31,095 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 159 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34718 deadline: 1681606531093, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:31,095 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:31,095 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34680 deadline: 1681606531093, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:31,095 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:31,095 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 161 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34702 deadline: 1681606531093, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:31,095 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:31,095 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34714 deadline: 1681606531093, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:31,297 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:31,297 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 161 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34718 deadline: 1681606531296, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:31,297 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:31,297 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34702 deadline: 1681606531296, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:31,298 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:31,298 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34680 deadline: 1681606531297, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:31,298 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:31,298 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34714 deadline: 1681606531297, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:31,392 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=348 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/.tmp/C/b6ec6a87ff8b445b94632270fc3614e5 2023-04-16 00:54:31,397 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/.tmp/A/0be1677bce94454eb2808230b3f443de as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/A/0be1677bce94454eb2808230b3f443de 2023-04-16 00:54:31,400 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/A/0be1677bce94454eb2808230b3f443de, entries=150, sequenceid=348, filesize=25.4 K 2023-04-16 00:54:31,401 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/.tmp/B/c7e598dcb45e40aea0c5899f5dd1c491 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/B/c7e598dcb45e40aea0c5899f5dd1c491 2023-04-16 00:54:31,405 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/B/c7e598dcb45e40aea0c5899f5dd1c491, entries=150, sequenceid=348, filesize=11.9 K 2023-04-16 00:54:31,406 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/.tmp/C/b6ec6a87ff8b445b94632270fc3614e5 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/C/b6ec6a87ff8b445b94632270fc3614e5 2023-04-16 00:54:31,410 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/C/b6ec6a87ff8b445b94632270fc3614e5, entries=150, sequenceid=348, filesize=11.9 K 2023-04-16 00:54:31,410 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=140.89 KB/144270 for bcfff279d7c9f90f5c57aef887110e5f in 1265ms, sequenceid=348, compaction requested=true 2023-04-16 00:54:31,410 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for bcfff279d7c9f90f5c57aef887110e5f: 2023-04-16 00:54:31,410 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f. 2023-04-16 00:54:31,410 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:54:31,410 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:54:31,411 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:31,411 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:54:31,411 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:54:31,412 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:31,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:31,412 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:31,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:31,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:31,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:31,413 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:31,413 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:54:31,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:31,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:31,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:31,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:31,414 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:54:31,414 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@b0801f7[Count = 0] remaining members to acquire global barrier 2023-04-16 00:54:31,414 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:54:31,414 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:31,416 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:31,416 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:31,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:31,416 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:54:31,416 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:54:31,416 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:54:31,416 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:31,416 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:54:31,418 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:54:31,418 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:31,418 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:54:31,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:31,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:31,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:31,418 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:54:31,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:31,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:31,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:31,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:31,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:31,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:31,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:31,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:54:31,420 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:54:31,420 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:54:31,420 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:54:31,421 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:54:31,421 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:54:31,421 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:54:31,422 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:31,422 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:31,422 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:31,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:31,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:31,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:31,422 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:54:31,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:31,423 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:54:31,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:31,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:54:31,423 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:31,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:31,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:31,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:31,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:31,424 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:31,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:31,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:31,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:31,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:31,429 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:31,429 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:54:31,429 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:31,429 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:54:31,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:54:31,430 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:54:31,429 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:54:31,430 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:54:31,430 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:54:31,429 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:31,430 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:31,430 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:54:31,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:54:31,430 DEBUG [Thread-984] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:54:31,430 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:31,430 DEBUG [Thread-984] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:54:31,430 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:31,430 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:31,530 DEBUG [Thread-984] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:54:31,531 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:54:31,532 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:54:31,533 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:54:31,534 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:54:31,534 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:54:31,534 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:54:31,534 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:54:31,534 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:54:31,534 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:31,534 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:31,537 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:54:31,537 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:31,537 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:54:31,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:54:31,537 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:31,537 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:54:31,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:31,537 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:31,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:54:31,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:31,538 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:54:31,538 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:54:31,538 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:54:31,538 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:54:31,538 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:54:31,538 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:54:31,538 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:54:31,538 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:54:31,538 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f. 2023-04-16 00:54:31,538 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f. started... 2023-04-16 00:54:31,539 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing bcfff279d7c9f90f5c57aef887110e5f 3/3 column families, dataSize=140.89 KB heapSize=381.47 KB 2023-04-16 00:54:31,539 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bcfff279d7c9f90f5c57aef887110e5f, store=A 2023-04-16 00:54:31,539 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 00:54:31,539 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bcfff279d7c9f90f5c57aef887110e5f, store=B 2023-04-16 00:54:31,539 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 00:54:31,539 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bcfff279d7c9f90f5c57aef887110e5f, store=C 2023-04-16 00:54:31,539 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 00:54:31,568 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304164b0ae11fe269429db93855bf9ecb3a69 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304164b0ae11fe269429db93855bf9ecb3a69 2023-04-16 00:54:31,601 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(9158): Flush requested on bcfff279d7c9f90f5c57aef887110e5f 2023-04-16 00:54:31,601 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f. as already flushing 2023-04-16 00:54:31,608 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:31,608 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 159 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34680 deadline: 1681606531603, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:31,608 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:31,608 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34718 deadline: 1681606531604, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:31,609 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:31,609 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34702 deadline: 1681606531605, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:31,611 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:31,611 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 161 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34714 deadline: 1681606531608, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:31,712 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:31,712 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 161 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34680 deadline: 1681606531709, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:31,713 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:31,713 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34718 deadline: 1681606531709, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:31,713 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:31,713 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34702 deadline: 1681606531710, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:31,713 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:31,713 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34714 deadline: 1681606531711, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:31,918 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:31,918 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34680 deadline: 1681606531913, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:31,919 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:31,919 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:31,919 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34718 deadline: 1681606531914, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:31,919 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34714 deadline: 1681606531914, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:31,919 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:31,919 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34702 deadline: 1681606531915, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:31,982 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=381, memsize=47.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/.tmp/A/4824237334444d00b911471b2a96ef53 2023-04-16 00:54:32,223 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:32,223 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34718 deadline: 1681606532221, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:32,223 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:32,223 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34702 deadline: 1681606532221, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:32,224 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:32,224 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34680 deadline: 1681606532221, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:32,224 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:32,224 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34714 deadline: 1681606532222, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:32,400 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=381 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/.tmp/B/0a647c4a83754901a9fd449ebbc8364f 2023-04-16 00:54:32,725 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:32,725 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34680 deadline: 1681606532725, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:32,729 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:32,729 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 175 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34702 deadline: 1681606532726, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:32,732 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:32,732 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34714 deadline: 1681606532728, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:32,732 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:32,732 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34718 deadline: 1681606532728, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:32,811 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=381 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/.tmp/C/463ab8e515bb48a9b12d7af4426426be 2023-04-16 00:54:32,815 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/.tmp/A/4824237334444d00b911471b2a96ef53 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/A/4824237334444d00b911471b2a96ef53 2023-04-16 00:54:32,818 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/A/4824237334444d00b911471b2a96ef53, entries=150, sequenceid=381, filesize=25.4 K 2023-04-16 00:54:32,819 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/.tmp/B/0a647c4a83754901a9fd449ebbc8364f as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/B/0a647c4a83754901a9fd449ebbc8364f 2023-04-16 00:54:32,823 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/B/0a647c4a83754901a9fd449ebbc8364f, entries=150, sequenceid=381, filesize=11.9 K 2023-04-16 00:54:32,824 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/.tmp/C/463ab8e515bb48a9b12d7af4426426be as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/C/463ab8e515bb48a9b12d7af4426426be 2023-04-16 00:54:32,827 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/C/463ab8e515bb48a9b12d7af4426426be, entries=150, sequenceid=381, filesize=11.9 K 2023-04-16 00:54:32,828 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~381.42 KB/390576, currentSize=46.96 KB/48090 for bcfff279d7c9f90f5c57aef887110e5f in 1289ms, sequenceid=381, compaction requested=true 2023-04-16 00:54:32,828 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for bcfff279d7c9f90f5c57aef887110e5f: 2023-04-16 00:54:32,828 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f. 2023-04-16 00:54:32,828 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:54:32,828 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:54:32,828 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:32,828 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:54:32,828 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:54:32,829 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:32,829 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:32,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:32,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:32,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:32,830 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:32,830 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:54:32,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:32,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:32,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:32,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:32,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:32,831 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:54:32,831 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@693b6967[Count = 0] remaining members to acquire global barrier 2023-04-16 00:54:32,831 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:54:32,831 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:32,832 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:32,832 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:32,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:32,832 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:54:32,832 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:54:32,832 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:54:32,832 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:32,832 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:54:32,833 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:32,833 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:54:32,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:32,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:32,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:32,833 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:54:32,834 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:54:32,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:32,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:32,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:32,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:32,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:32,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:32,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:32,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:54:32,836 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:54:32,836 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:54:32,836 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:54:32,836 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:54:32,836 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:54:32,836 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:54:32,838 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:32,838 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:32,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:32,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:32,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:32,838 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:32,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:32,838 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:54:32,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:32,839 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:32,839 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:54:32,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:54:32,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:32,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:32,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:32,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:32,840 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:32,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:32,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:32,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:32,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:32,842 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:54:32,842 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:54:32,842 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:54:32,842 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:54:32,842 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:32,842 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:54:32,842 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:54:32,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:54:32,842 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:32,842 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:32,842 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:32,842 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:32,842 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:32,842 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:32,843 DEBUG [Thread-984] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:54:32,843 DEBUG [Thread-984] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:54:32,843 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:54:32,843 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:54:32,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:54:32,943 DEBUG [Thread-984] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:54:32,943 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:54:32,945 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:54:32,946 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:54:32,946 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:54:32,946 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:54:32,946 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:54:32,946 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:54:32,946 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:54:32,947 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:32,947 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:32,948 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:32,948 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:54:32,948 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:54:32,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:54:32,948 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:32,948 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:54:32,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:32,949 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:32,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:54:32,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:32,949 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:54:32,949 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:54:32,949 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:54:32,950 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:54:32,950 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:54:32,950 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:54:32,950 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:54:32,950 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:54:32,950 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f. 2023-04-16 00:54:32,950 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f. started... 2023-04-16 00:54:32,950 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing bcfff279d7c9f90f5c57aef887110e5f 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-16 00:54:32,950 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bcfff279d7c9f90f5c57aef887110e5f, store=A 2023-04-16 00:54:32,950 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:54:32,951 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bcfff279d7c9f90f5c57aef887110e5f, store=B 2023-04-16 00:54:32,951 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:54:32,951 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bcfff279d7c9f90f5c57aef887110e5f, store=C 2023-04-16 00:54:32,951 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:54:32,965 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230416ce31c3f3754346928bda7ba990b835e0 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230416ce31c3f3754346928bda7ba990b835e0 2023-04-16 00:54:32,970 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=391, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/.tmp/A/12a77a02ea8c4ca39a9ee6fd692f7bf5 2023-04-16 00:54:32,996 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=391 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/.tmp/B/4a47a809cdf045b0bff0c9fd7cc0ce3a 2023-04-16 00:54:33,408 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=391 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/.tmp/C/2c82f146691c43a09502c2c522f8b7cf 2023-04-16 00:54:33,412 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/.tmp/A/12a77a02ea8c4ca39a9ee6fd692f7bf5 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/A/12a77a02ea8c4ca39a9ee6fd692f7bf5 2023-04-16 00:54:33,416 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/A/12a77a02ea8c4ca39a9ee6fd692f7bf5, entries=150, sequenceid=391, filesize=25.4 K 2023-04-16 00:54:33,417 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/.tmp/B/4a47a809cdf045b0bff0c9fd7cc0ce3a as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/B/4a47a809cdf045b0bff0c9fd7cc0ce3a 2023-04-16 00:54:33,420 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/B/4a47a809cdf045b0bff0c9fd7cc0ce3a, entries=150, sequenceid=391, filesize=11.9 K 2023-04-16 00:54:33,421 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/.tmp/C/2c82f146691c43a09502c2c522f8b7cf as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/C/2c82f146691c43a09502c2c522f8b7cf 2023-04-16 00:54:33,425 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/C/2c82f146691c43a09502c2c522f8b7cf, entries=150, sequenceid=391, filesize=11.9 K 2023-04-16 00:54:33,426 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=0 B/0 for bcfff279d7c9f90f5c57aef887110e5f in 476ms, sequenceid=391, compaction requested=true 2023-04-16 00:54:33,426 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for bcfff279d7c9f90f5c57aef887110e5f: 2023-04-16 00:54:33,426 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f. 2023-04-16 00:54:33,426 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:54:33,426 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:54:33,426 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:33,426 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:54:33,426 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:54:33,428 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:33,428 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:33,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:33,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:33,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:33,428 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:33,428 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:54:33,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:33,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:33,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:33,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:33,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:33,430 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:54:33,430 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5cf097a0[Count = 0] remaining members to acquire global barrier 2023-04-16 00:54:33,430 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:54:33,430 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:33,432 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:33,432 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:33,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:33,432 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:54:33,432 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:33,432 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:54:33,432 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:54:33,432 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:54:33,434 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:54:33,434 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:33,434 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:54:33,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:33,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:33,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:33,434 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:54:33,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:33,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:33,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:33,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:33,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:33,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:33,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:33,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:54:33,436 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:54:33,436 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:54:33,436 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:54:33,436 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:54:33,436 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:54:33,437 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:54:33,439 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:33,439 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:33,439 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:33,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:33,439 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:54:33,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:33,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:33,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:33,439 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:54:33,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:54:33,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:33,439 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:33,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:33,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:33,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:33,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:33,440 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:33,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:33,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:33,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:33,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:33,442 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:33,443 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:33,443 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:33,443 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:33,443 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:33,443 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:54:33,443 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:33,443 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:54:33,443 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:54:33,443 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:54:33,443 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:54:33,443 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:54:33,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:54:33,443 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:54:33,443 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:33,443 DEBUG [Thread-984] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:54:33,443 DEBUG [Thread-984] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:54:33,443 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:54:33,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:54:33,543 DEBUG [Thread-984] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:54:33,544 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:54:33,546 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:54:33,547 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:54:33,547 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:54:33,547 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:54:33,547 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:54:33,547 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:54:33,547 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:54:33,548 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:33,548 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:33,552 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:33,552 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:54:33,552 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:54:33,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:54:33,552 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:33,552 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:54:33,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:33,553 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:33,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:54:33,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:33,553 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:54:33,554 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:54:33,554 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:54:33,555 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:54:33,555 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:54:33,555 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:54:33,555 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:54:33,555 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f. 2023-04-16 00:54:33,555 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f. started... 2023-04-16 00:54:33,555 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for bcfff279d7c9f90f5c57aef887110e5f: 2023-04-16 00:54:33,555 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f. 2023-04-16 00:54:33,555 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:54:33,555 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:54:33,555 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:33,555 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:54:33,555 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:54:33,557 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:33,557 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:33,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:33,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:33,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:33,557 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:33,557 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:54:33,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:33,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:33,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:33,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:33,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:33,559 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:54:33,559 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4797ee4a[Count = 0] remaining members to acquire global barrier 2023-04-16 00:54:33,559 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:54:33,559 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:33,560 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:33,560 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:33,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:33,560 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:54:33,560 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:54:33,560 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:54:33,561 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:33,561 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:54:33,562 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:33,562 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:54:33,562 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:54:33,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:33,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:33,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:33,562 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:54:33,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:33,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:33,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:33,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:33,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:33,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:33,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:33,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:54:33,565 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:54:33,565 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:54:33,565 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:54:33,565 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:54:33,565 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:54:33,565 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:54:33,567 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:33,567 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:33,567 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:33,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:33,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:33,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:33,567 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:54:33,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:33,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:33,567 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:33,567 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:54:33,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:54:33,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:33,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:33,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:33,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:33,568 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:33,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:33,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:33,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:33,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:33,571 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:33,571 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:33,571 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:33,571 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:54:33,571 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:33,571 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:33,571 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:33,571 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:33,571 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:54:33,571 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:54:33,571 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:54:33,571 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:54:33,571 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:54:33,571 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:54:33,571 DEBUG [Thread-984] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:54:33,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:54:33,571 DEBUG [Thread-984] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:54:33,572 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:54:33,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:54:33,671 DEBUG [Thread-984] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:54:33,672 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:54:33,674 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:54:33,675 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:54:33,675 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:54:33,675 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:54:33,675 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:54:33,675 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:54:33,675 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:54:33,675 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:33,676 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:33,677 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:54:33,677 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:33,677 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:54:33,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:54:33,677 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:33,677 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:54:33,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:33,678 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:33,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:54:33,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:33,678 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:54:33,678 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:54:33,678 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:54:33,678 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:54:33,678 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:54:33,679 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:54:33,679 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:54:33,679 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:54:33,679 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f. 2023-04-16 00:54:33,679 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f. started... 2023-04-16 00:54:33,679 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for bcfff279d7c9f90f5c57aef887110e5f: 2023-04-16 00:54:33,679 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f. 2023-04-16 00:54:33,679 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:54:33,679 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:54:33,679 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:33,679 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:54:33,679 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:54:33,681 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:33,681 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:33,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:33,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:33,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:33,682 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:33,682 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:54:33,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:33,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:33,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:33,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:33,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:33,683 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:54:33,683 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@cdb58f6[Count = 0] remaining members to acquire global barrier 2023-04-16 00:54:33,683 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:54:33,683 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:33,685 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:33,685 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:33,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:33,685 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:54:33,685 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:54:33,685 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:33,685 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:54:33,685 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:54:33,687 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:54:33,687 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:33,687 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:54:33,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:33,687 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:54:33,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:33,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:33,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:33,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:33,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:33,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:33,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:33,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:33,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:33,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:54:33,689 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:54:33,689 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:54:33,690 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:54:33,690 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:54:33,690 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:54:33,690 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:54:33,691 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:33,691 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:33,692 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:54:33,692 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:33,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:33,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:33,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:33,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:33,692 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:54:33,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:54:33,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:33,692 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:33,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:33,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:33,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:33,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:33,694 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:33,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:33,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:33,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:33,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:33,698 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:33,698 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:33,698 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:33,698 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:33,698 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:33,698 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:33,698 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:54:33,698 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:54:33,698 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:54:33,698 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:54:33,698 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:54:33,698 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:54:33,698 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:33,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:54:33,698 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:54:33,698 DEBUG [Thread-984] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:54:33,699 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:54:33,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:54:33,699 DEBUG [Thread-984] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:54:33,761 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(9158): Flush requested on bcfff279d7c9f90f5c57aef887110e5f 2023-04-16 00:54:33,761 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing bcfff279d7c9f90f5c57aef887110e5f 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-16 00:54:33,762 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bcfff279d7c9f90f5c57aef887110e5f, store=A 2023-04-16 00:54:33,762 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:54:33,762 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bcfff279d7c9f90f5c57aef887110e5f, store=B 2023-04-16 00:54:33,762 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:54:33,762 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bcfff279d7c9f90f5c57aef887110e5f, store=C 2023-04-16 00:54:33,762 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:54:33,787 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14144, totalHeapSize=32376, min timestamp=1681606473774, max timestamp=1681606473785Num uniques 859; 2023-04-16 00:54:33,787 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=14144, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-16 00:54:33,787 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14144, totalHeapSize=32376, min timestamp=1681606473774, max timestamp=1681606473785Num uniques 859; 2023-04-16 00:54:33,787 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14144, totalHeapSize=32376, min timestamp=1681606473774, max timestamp=1681606473785Num uniques 859; 2023-04-16 00:54:33,787 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=14144, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-16 00:54:33,787 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=14144, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-16 00:54:33,799 DEBUG [Thread-984] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:54:33,799 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:54:33,801 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:54:33,802 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:54:33,802 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:54:33,802 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:54:33,802 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:54:33,802 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:54:33,802 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:54:33,802 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:33,802 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:33,804 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:33,804 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:54:33,804 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:54:33,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:54:33,804 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:33,804 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:54:33,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:33,805 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:33,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:54:33,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:33,805 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:54:33,805 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:54:33,805 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:54:33,805 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:54:33,805 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:54:33,805 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:54:33,806 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:54:33,806 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:54:33,806 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f. 2023-04-16 00:54:33,806 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f. started... 2023-04-16 00:54:33,806 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f. as already flushing 2023-04-16 00:54:33,806 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f. 2023-04-16 00:54:33,806 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:54:33,806 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:54:33,806 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:33,806 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:54:33,806 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:54:33,807 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:33,807 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:33,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:33,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:33,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:33,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:33,808 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:33,808 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:54:33,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:33,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:33,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:33,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:33,809 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:54:33,809 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3e99785b[Count = 0] remaining members to acquire global barrier 2023-04-16 00:54:33,809 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:54:33,810 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:33,810 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:33,810 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 181 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34714 deadline: 1681606533805, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:33,810 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:33,810 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 187 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34702 deadline: 1681606533805, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:33,811 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:33,811 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:33,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:33,811 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:54:33,811 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:54:33,811 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:54:33,811 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:33,811 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:54:33,813 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:33,813 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 179 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34680 deadline: 1681606533810, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:33,813 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:33,813 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:33,813 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:54:33,813 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:54:33,813 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 185 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34718 deadline: 1681606533810, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:33,813 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:54:33,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:33,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:33,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:33,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:33,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:33,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:33,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:33,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:33,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:33,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:33,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:54:33,816 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:54:33,816 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:54:33,816 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:54:33,816 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:54:33,816 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:54:33,816 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:54:33,818 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:33,818 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:33,818 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:33,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:33,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:33,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:33,818 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:54:33,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:33,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:33,819 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:54:33,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:54:33,819 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:33,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:33,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:33,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:33,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:33,820 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:33,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:33,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:33,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:33,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:33,830 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:33,830 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:33,830 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:33,830 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:54:33,830 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:54:33,830 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:33,830 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:54:33,830 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:54:33,830 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:54:33,830 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:54:33,830 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:33,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:54:33,830 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:33,830 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:33,830 DEBUG [Thread-984] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:54:33,830 DEBUG [Thread-984] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:54:33,831 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:54:33,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:54:33,912 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:33,913 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 183 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34714 deadline: 1681606533911, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:33,913 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:33,913 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 189 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34702 deadline: 1681606533911, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:33,919 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:33,919 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 181 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34680 deadline: 1681606533914, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:33,919 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:33,919 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 187 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34718 deadline: 1681606533914, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:33,931 DEBUG [Thread-984] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:54:33,931 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:54:33,932 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:54:33,934 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:54:33,934 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:54:33,934 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:54:33,934 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:54:33,934 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:54:33,934 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:54:33,934 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:33,934 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:33,936 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:33,936 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:54:33,936 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:54:33,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:54:33,936 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:33,936 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:54:33,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:33,937 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:33,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:54:33,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:33,938 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:54:33,938 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:54:33,938 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:54:33,938 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:54:33,938 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:54:33,938 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:54:33,939 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:54:33,939 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:54:33,939 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f. 2023-04-16 00:54:33,939 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f. started... 2023-04-16 00:54:33,939 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f. as already flushing 2023-04-16 00:54:33,939 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f. 2023-04-16 00:54:33,939 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:54:33,939 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:54:33,939 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:33,939 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:54:33,939 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:54:33,941 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:33,941 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:33,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:33,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:33,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:33,941 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:33,941 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:54:33,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:33,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:33,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:33,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:33,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:33,943 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:54:33,943 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@660ca5bf[Count = 0] remaining members to acquire global barrier 2023-04-16 00:54:33,943 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:54:33,943 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:33,946 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:33,946 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:33,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:33,946 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:54:33,946 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:54:33,946 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:33,947 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:54:33,947 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:54:33,948 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:54:33,948 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:54:33,948 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:33,948 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:54:33,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:33,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:33,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:33,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:33,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:33,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:33,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:33,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:33,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:33,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:33,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:54:33,951 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:54:33,951 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:54:33,951 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:54:33,951 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:54:33,951 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:54:33,951 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:54:33,952 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:33,952 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:33,952 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:54:33,952 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:33,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:33,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:33,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:33,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:33,953 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:54:33,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:33,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:54:33,953 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:33,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:33,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:33,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:33,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:33,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:33,954 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:33,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:33,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:33,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:33,958 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:54:33,958 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:54:33,958 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:54:33,958 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:54:33,958 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:54:33,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:54:33,958 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:54:33,958 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:54:33,958 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:33,958 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:33,958 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:33,958 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:33,958 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:33,958 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:54:33,958 DEBUG [Thread-984] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:54:33,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:54:33,958 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:33,959 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:33,958 DEBUG [Thread-984] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:54:34,059 DEBUG [Thread-984] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:54:34,059 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:54:34,061 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:54:34,062 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:54:34,062 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:54:34,062 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:54:34,062 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:54:34,062 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:54:34,062 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:54:34,062 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:34,062 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:34,064 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:34,064 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:54:34,064 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:54:34,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:54:34,064 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:34,064 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:54:34,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:34,065 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:34,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:54:34,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:34,065 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:54:34,065 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:54:34,065 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:54:34,065 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:54:34,065 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:54:34,065 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:54:34,065 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:54:34,065 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:54:34,066 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f. 2023-04-16 00:54:34,066 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f. started... 2023-04-16 00:54:34,066 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f. as already flushing 2023-04-16 00:54:34,066 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f. 2023-04-16 00:54:34,066 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:54:34,066 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:54:34,066 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:34,066 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:54:34,066 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:54:34,067 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:34,067 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:34,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:34,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:34,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:34,068 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:34,068 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:54:34,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:34,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:34,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:34,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:34,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:34,069 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:54:34,069 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1e11e28a[Count = 0] remaining members to acquire global barrier 2023-04-16 00:54:34,069 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:54:34,069 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:34,070 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:34,071 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:34,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:34,071 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:34,071 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:54:34,071 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:54:34,071 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:54:34,071 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:54:34,077 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:34,077 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:54:34,077 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:54:34,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:34,077 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:54:34,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:34,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:34,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:34,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:34,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:34,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:34,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:34,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:34,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:34,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:54:34,079 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:54:34,079 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:54:34,080 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:54:34,080 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:54:34,080 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:54:34,080 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:54:34,081 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:34,081 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:34,081 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:34,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:34,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:34,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:34,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:34,081 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:54:34,081 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:54:34,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:34,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:54:34,082 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:34,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:34,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:34,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:34,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:34,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:34,083 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:34,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:34,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:34,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:34,086 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:34,086 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:54:34,086 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:34,086 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:54:34,086 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:54:34,086 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:54:34,086 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:54:34,086 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:34,086 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:54:34,086 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:34,086 DEBUG [Thread-984] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:54:34,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:54:34,086 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:54:34,086 DEBUG [Thread-984] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:54:34,086 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:34,087 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:34,087 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:34,087 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:54:34,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:54:34,115 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:34,116 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 185 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34714 deadline: 1681606534114, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:34,116 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:34,116 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 191 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34702 deadline: 1681606534114, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:34,126 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:34,126 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 183 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34680 deadline: 1681606534121, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:34,126 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:34,126 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 189 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34718 deadline: 1681606534121, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:34,176 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230416bdcdc451a7e5498594cd9c2a8d5497c8 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230416bdcdc451a7e5498594cd9c2a8d5497c8 2023-04-16 00:54:34,182 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=404, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/.tmp/A/e0b09a14eb324581b9ca2ec144e1a33e 2023-04-16 00:54:34,187 DEBUG [Thread-984] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:54:34,187 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:54:34,190 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:54:34,192 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:54:34,192 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:54:34,193 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:54:34,193 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:54:34,193 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:54:34,193 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:54:34,195 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:34,195 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:34,196 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:34,196 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:54:34,196 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:54:34,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:54:34,197 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:34,197 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:54:34,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:34,197 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:34,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:54:34,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:34,197 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:54:34,209 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:54:34,209 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:54:34,209 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:54:34,209 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:54:34,209 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:54:34,209 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:54:34,209 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:54:34,210 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f. 2023-04-16 00:54:34,210 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f. started... 2023-04-16 00:54:34,210 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f. as already flushing 2023-04-16 00:54:34,210 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f. 2023-04-16 00:54:34,210 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:54:34,210 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:54:34,210 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:34,216 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:54:34,216 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:54:34,218 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:34,218 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:34,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:34,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:34,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:34,219 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:34,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:34,219 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:54:34,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:34,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:34,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:34,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:34,220 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:54:34,220 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@48db1556[Count = 0] remaining members to acquire global barrier 2023-04-16 00:54:34,220 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:54:34,220 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:34,221 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:34,222 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:34,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:34,222 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:54:34,222 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:54:34,222 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:54:34,222 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:34,222 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:54:34,224 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:34,224 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:54:34,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:34,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:34,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:34,224 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:54:34,224 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:54:34,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:34,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:34,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:34,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:34,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:34,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:34,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:34,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:54:34,228 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:54:34,228 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:54:34,228 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:54:34,228 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:54:34,229 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:54:34,229 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:54:34,230 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:34,230 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:54:34,230 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:34,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:34,230 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:34,231 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:54:34,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:54:34,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:34,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:34,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:34,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:34,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:34,232 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:34,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:34,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:34,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:34,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:34,233 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:34,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:34,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:34,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:34,236 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:54:34,236 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:54:34,236 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:54:34,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:54:34,237 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:34,237 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:34,237 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:34,237 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:34,237 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:34,237 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:34,237 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:34,237 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:54:34,237 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:54:34,237 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:54:34,237 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:54:34,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:54:34,237 DEBUG [Thread-984] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:54:34,237 DEBUG [Thread-984] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:54:34,249 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=404 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/.tmp/B/2769b340de12499c9508aa15412f083a 2023-04-16 00:54:34,268 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=404 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/.tmp/C/224fa70428f84145a9412d779925f5b2 2023-04-16 00:54:34,274 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/.tmp/A/e0b09a14eb324581b9ca2ec144e1a33e as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/A/e0b09a14eb324581b9ca2ec144e1a33e 2023-04-16 00:54:34,279 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/A/e0b09a14eb324581b9ca2ec144e1a33e, entries=400, sequenceid=404, filesize=59.8 K 2023-04-16 00:54:34,280 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/.tmp/B/2769b340de12499c9508aa15412f083a as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/B/2769b340de12499c9508aa15412f083a 2023-04-16 00:54:34,285 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/B/2769b340de12499c9508aa15412f083a, entries=150, sequenceid=404, filesize=11.9 K 2023-04-16 00:54:34,286 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/.tmp/C/224fa70428f84145a9412d779925f5b2 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/C/224fa70428f84145a9412d779925f5b2 2023-04-16 00:54:34,293 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/C/224fa70428f84145a9412d779925f5b2, entries=150, sequenceid=404, filesize=11.9 K 2023-04-16 00:54:34,297 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=140.89 KB/144270 for bcfff279d7c9f90f5c57aef887110e5f in 536ms, sequenceid=404, compaction requested=true 2023-04-16 00:54:34,297 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for bcfff279d7c9f90f5c57aef887110e5f: 2023-04-16 00:54:34,297 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 00:54:34,297 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-16 00:54:34,297 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-16 00:54:34,297 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 00:54:34,297 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 00:54:34,300 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 192064 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-16 00:54:34,300 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HStore(1912): bcfff279d7c9f90f5c57aef887110e5f/A is initiating minor compaction (all files) 2023-04-16 00:54:34,300 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of bcfff279d7c9f90f5c57aef887110e5f/A in TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f. 2023-04-16 00:54:34,300 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/A/c274a82a5dfb44d3a9d5d0f8b55f9678, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/A/294c519045464f2ebd5223681bbdaa69, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/A/0be1677bce94454eb2808230b3f443de, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/A/4824237334444d00b911471b2a96ef53, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/A/12a77a02ea8c4ca39a9ee6fd692f7bf5, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/A/e0b09a14eb324581b9ca2ec144e1a33e] into tmpdir=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/.tmp, totalSize=187.6 K 2023-04-16 00:54:34,301 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 73696 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-16 00:54:34,301 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HStore(1912): bcfff279d7c9f90f5c57aef887110e5f/B is initiating minor compaction (all files) 2023-04-16 00:54:34,301 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HRegion(2259): Starting compaction of bcfff279d7c9f90f5c57aef887110e5f/B in TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f. 2023-04-16 00:54:34,301 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/B/21b806fc0f7f41c2b61745a3181c339e, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/B/640431c713c94b3bb44acca27ba0b0ca, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/B/c7e598dcb45e40aea0c5899f5dd1c491, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/B/0a647c4a83754901a9fd449ebbc8364f, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/B/4a47a809cdf045b0bff0c9fd7cc0ce3a, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/B/2769b340de12499c9508aa15412f083a] into tmpdir=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/.tmp, totalSize=72.0 K 2023-04-16 00:54:34,302 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.Compactor(207): Compacting c274a82a5dfb44d3a9d5d0f8b55f9678, keycount=150, bloomtype=ROW, size=26.0 K, encoding=NONE, compression=NONE, seqNum=299, earliestPutTs=1681606468148 2023-04-16 00:54:34,302 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting 21b806fc0f7f41c2b61745a3181c339e, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=299, earliestPutTs=1681606468148 2023-04-16 00:54:34,302 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.Compactor(207): Compacting 294c519045464f2ebd5223681bbdaa69, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=335, earliestPutTs=1681606468208 2023-04-16 00:54:34,302 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting 640431c713c94b3bb44acca27ba0b0ca, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=335, earliestPutTs=1681606468208 2023-04-16 00:54:34,302 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.Compactor(207): Compacting 0be1677bce94454eb2808230b3f443de, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=348, earliestPutTs=1681606468833 2023-04-16 00:54:34,303 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting c7e598dcb45e40aea0c5899f5dd1c491, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=348, earliestPutTs=1681606468833 2023-04-16 00:54:34,303 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.Compactor(207): Compacting 4824237334444d00b911471b2a96ef53, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=381, earliestPutTs=1681606470981 2023-04-16 00:54:34,304 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting 0a647c4a83754901a9fd449ebbc8364f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=381, earliestPutTs=1681606470981 2023-04-16 00:54:34,304 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.Compactor(207): Compacting 12a77a02ea8c4ca39a9ee6fd692f7bf5, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=391, earliestPutTs=1681606471602 2023-04-16 00:54:34,305 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting 4a47a809cdf045b0bff0c9fd7cc0ce3a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=391, earliestPutTs=1681606471602 2023-04-16 00:54:34,305 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.Compactor(207): Compacting e0b09a14eb324581b9ca2ec144e1a33e, keycount=400, bloomtype=ROW, size=59.8 K, encoding=NONE, compression=NONE, seqNum=404, earliestPutTs=1681606473746 2023-04-16 00:54:34,306 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting 2769b340de12499c9508aa15412f083a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=404, earliestPutTs=1681606473749 2023-04-16 00:54:34,321 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] throttle.PressureAwareThroughputController(145): bcfff279d7c9f90f5c57aef887110e5f#B#compaction#196 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-16 00:54:34,327 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] throttle.PressureAwareThroughputController(145): bcfff279d7c9f90f5c57aef887110e5f#A#compaction#195 average throughput is 1.97 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-16 00:54:34,337 DEBUG [Thread-984] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:54:34,338 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:54:34,339 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:54:34,359 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/.tmp/B/d921f30e70c84bed960a37d28b43b8c4 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/B/d921f30e70c84bed960a37d28b43b8c4 2023-04-16 00:54:34,360 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:54:34,360 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:54:34,360 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:54:34,360 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:54:34,360 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:54:34,360 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:54:34,361 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:34,361 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:34,363 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:34,363 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:54:34,363 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:54:34,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:54:34,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:34,364 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:34,364 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:54:34,364 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:34,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:54:34,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:34,365 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:54:34,365 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:54:34,365 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:54:34,365 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:54:34,365 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:54:34,365 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:54:34,366 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in bcfff279d7c9f90f5c57aef887110e5f/B of bcfff279d7c9f90f5c57aef887110e5f into d921f30e70c84bed960a37d28b43b8c4(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 00:54:34,366 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for bcfff279d7c9f90f5c57aef887110e5f: 2023-04-16 00:54:34,366 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f., storeName=bcfff279d7c9f90f5c57aef887110e5f/B, priority=10, startTime=1681606474297; duration=0sec 2023-04-16 00:54:34,367 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 00:54:34,367 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-16 00:54:34,372 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:54:34,372 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:54:34,372 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f. 2023-04-16 00:54:34,372 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f. started... 2023-04-16 00:54:34,372 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing bcfff279d7c9f90f5c57aef887110e5f 3/3 column families, dataSize=140.89 KB heapSize=390.84 KB 2023-04-16 00:54:34,373 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 73696 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-16 00:54:34,373 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HStore(1912): bcfff279d7c9f90f5c57aef887110e5f/C is initiating minor compaction (all files) 2023-04-16 00:54:34,373 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HRegion(2259): Starting compaction of bcfff279d7c9f90f5c57aef887110e5f/C in TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f. 2023-04-16 00:54:34,373 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/C/54170cd1ceda4274a259fe41ee4da8f8, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/C/621791b9c43346cdbf24e4d25984a701, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/C/b6ec6a87ff8b445b94632270fc3614e5, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/C/463ab8e515bb48a9b12d7af4426426be, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/C/2c82f146691c43a09502c2c522f8b7cf, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/C/224fa70428f84145a9412d779925f5b2] into tmpdir=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/.tmp, totalSize=72.0 K 2023-04-16 00:54:34,373 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting 54170cd1ceda4274a259fe41ee4da8f8, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=299, earliestPutTs=1681606468148 2023-04-16 00:54:34,374 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting 621791b9c43346cdbf24e4d25984a701, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=335, earliestPutTs=1681606468208 2023-04-16 00:54:34,374 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting b6ec6a87ff8b445b94632270fc3614e5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=348, earliestPutTs=1681606468833 2023-04-16 00:54:34,374 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bcfff279d7c9f90f5c57aef887110e5f, store=A 2023-04-16 00:54:34,375 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting 463ab8e515bb48a9b12d7af4426426be, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=381, earliestPutTs=1681606470981 2023-04-16 00:54:34,375 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 00:54:34,375 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bcfff279d7c9f90f5c57aef887110e5f, store=B 2023-04-16 00:54:34,375 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 00:54:34,375 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting 2c82f146691c43a09502c2c522f8b7cf, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=391, earliestPutTs=1681606471602 2023-04-16 00:54:34,375 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bcfff279d7c9f90f5c57aef887110e5f, store=C 2023-04-16 00:54:34,375 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 00:54:34,375 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting 224fa70428f84145a9412d779925f5b2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=404, earliestPutTs=1681606473749 2023-04-16 00:54:34,407 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] throttle.PressureAwareThroughputController(145): bcfff279d7c9f90f5c57aef887110e5f#C#compaction#197 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-16 00:54:34,421 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f. as already flushing 2023-04-16 00:54:34,422 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(9158): Flush requested on bcfff279d7c9f90f5c57aef887110e5f 2023-04-16 00:54:34,426 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230416610afa9caefe4f93866d48308c418c91 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230416610afa9caefe4f93866d48308c418c91 2023-04-16 00:54:34,431 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/.tmp/C/68c6722fdd444523ac341216c11dcdc4 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/C/68c6722fdd444523ac341216c11dcdc4 2023-04-16 00:54:34,436 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in bcfff279d7c9f90f5c57aef887110e5f/C of bcfff279d7c9f90f5c57aef887110e5f into 68c6722fdd444523ac341216c11dcdc4(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 00:54:34,436 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for bcfff279d7c9f90f5c57aef887110e5f: 2023-04-16 00:54:34,436 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f., storeName=bcfff279d7c9f90f5c57aef887110e5f/C, priority=10, startTime=1681606474297; duration=0sec 2023-04-16 00:54:34,436 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 00:54:34,437 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:34,437 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:34,437 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 189 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34714 deadline: 1681606534429, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:34,437 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 186 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34680 deadline: 1681606534431, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:34,437 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:34,437 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 192 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34718 deadline: 1681606534435, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:34,450 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:34,450 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 196 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34702 deadline: 1681606534439, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:34,539 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:34,539 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 188 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34680 deadline: 1681606534538, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:34,539 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:34,539 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 191 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34714 deadline: 1681606534538, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:34,541 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:34,541 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 194 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34718 deadline: 1681606534540, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:34,554 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:34,555 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 198 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34702 deadline: 1681606534551, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:34,740 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:34,740 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 193 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34714 deadline: 1681606534740, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:34,743 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:34,743 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 190 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34680 deadline: 1681606534741, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:34,743 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:34,744 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 196 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34718 deadline: 1681606534743, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:34,759 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:34,760 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 200 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34702 deadline: 1681606534756, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:34,814 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/.tmp/A/0eaeed5ba5f34552934498a46e2aef33 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/A/0eaeed5ba5f34552934498a46e2aef33 2023-04-16 00:54:34,818 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in bcfff279d7c9f90f5c57aef887110e5f/A of bcfff279d7c9f90f5c57aef887110e5f into 0eaeed5ba5f34552934498a46e2aef33(size=26.2 K), total size for store is 26.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 00:54:34,818 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for bcfff279d7c9f90f5c57aef887110e5f: 2023-04-16 00:54:34,818 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f., storeName=bcfff279d7c9f90f5c57aef887110e5f/A, priority=10, startTime=1681606474297; duration=0sec 2023-04-16 00:54:34,818 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 00:54:34,842 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=440, memsize=47.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/.tmp/A/664e173ad4404213bab38d250b20769c 2023-04-16 00:54:34,853 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=440 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/.tmp/B/40c2c459df1a4c26b6277e187e31a9cc 2023-04-16 00:54:35,046 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:35,046 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 195 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34714 deadline: 1681606535043, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:35,047 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:35,047 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 198 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34718 deadline: 1681606535045, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:35,047 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:35,047 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 192 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34680 deadline: 1681606535046, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:35,063 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:35,063 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 202 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34702 deadline: 1681606535062, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:35,265 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=440 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/.tmp/C/0f00134bae7c4109a12dd915c06cdbc0 2023-04-16 00:54:35,269 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/.tmp/A/664e173ad4404213bab38d250b20769c as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/A/664e173ad4404213bab38d250b20769c 2023-04-16 00:54:35,273 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/A/664e173ad4404213bab38d250b20769c, entries=150, sequenceid=440, filesize=25.4 K 2023-04-16 00:54:35,274 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/.tmp/B/40c2c459df1a4c26b6277e187e31a9cc as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/B/40c2c459df1a4c26b6277e187e31a9cc 2023-04-16 00:54:35,277 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/B/40c2c459df1a4c26b6277e187e31a9cc, entries=150, sequenceid=440, filesize=11.9 K 2023-04-16 00:54:35,278 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/.tmp/C/0f00134bae7c4109a12dd915c06cdbc0 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/C/0f00134bae7c4109a12dd915c06cdbc0 2023-04-16 00:54:35,281 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/C/0f00134bae7c4109a12dd915c06cdbc0, entries=150, sequenceid=440, filesize=11.9 K 2023-04-16 00:54:35,282 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~390.80 KB/400176, currentSize=46.96 KB/48090 for bcfff279d7c9f90f5c57aef887110e5f in 910ms, sequenceid=440, compaction requested=false 2023-04-16 00:54:35,282 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for bcfff279d7c9f90f5c57aef887110e5f: 2023-04-16 00:54:35,282 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f. 2023-04-16 00:54:35,282 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:54:35,282 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:54:35,282 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:35,282 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:54:35,282 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:54:35,284 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:35,284 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:35,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:35,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:35,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:35,284 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:35,284 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:54:35,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:35,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:35,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:35,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:35,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:35,286 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:54:35,286 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3d75f7b6[Count = 0] remaining members to acquire global barrier 2023-04-16 00:54:35,286 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:54:35,286 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:35,287 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:35,287 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:35,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:35,287 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:54:35,287 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:54:35,287 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:54:35,287 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:35,288 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:54:35,297 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:35,297 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:54:35,297 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:54:35,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:35,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:35,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:35,297 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:54:35,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:35,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:35,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:35,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:35,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:35,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:35,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:35,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:54:35,300 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:54:35,300 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:54:35,300 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:54:35,300 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:54:35,300 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:54:35,300 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:54:35,301 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:35,302 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:35,302 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:35,302 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:54:35,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:35,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:35,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:35,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:35,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:35,302 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:54:35,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:54:35,302 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:35,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:35,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:35,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:35,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:35,303 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:35,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:35,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:35,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:35,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:35,306 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:35,306 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:54:35,306 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:35,306 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:54:35,306 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:54:35,306 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:54:35,306 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:54:35,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:54:35,306 DEBUG [Thread-984] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:54:35,306 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:54:35,306 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:54:35,306 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:35,306 DEBUG [Thread-984] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:54:35,307 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:35,307 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:35,307 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:35,307 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:35,307 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:54:35,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:54:35,407 DEBUG [Thread-984] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:54:35,407 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:54:35,409 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:54:35,410 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:54:35,410 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:54:35,410 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:54:35,410 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:54:35,410 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:54:35,410 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:54:35,410 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:35,410 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:35,412 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:54:35,412 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:35,412 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:54:35,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:54:35,412 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:35,412 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:54:35,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:35,412 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:35,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:54:35,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:35,413 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:54:35,413 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:54:35,413 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:54:35,413 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:54:35,413 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:54:35,413 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:54:35,413 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:54:35,413 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:54:35,413 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f. 2023-04-16 00:54:35,413 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f. started... 2023-04-16 00:54:35,414 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing bcfff279d7c9f90f5c57aef887110e5f 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-16 00:54:35,414 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bcfff279d7c9f90f5c57aef887110e5f, store=A 2023-04-16 00:54:35,414 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:54:35,414 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bcfff279d7c9f90f5c57aef887110e5f, store=B 2023-04-16 00:54:35,414 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:54:35,414 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bcfff279d7c9f90f5c57aef887110e5f, store=C 2023-04-16 00:54:35,414 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:54:35,549 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(9158): Flush requested on bcfff279d7c9f90f5c57aef887110e5f 2023-04-16 00:54:35,549 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f. as already flushing 2023-04-16 00:54:35,570 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681606475556, max timestamp=1681606475568Num uniques 859; 2023-04-16 00:54:35,570 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681606475556, max timestamp=1681606475568Num uniques 859; 2023-04-16 00:54:35,570 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-16 00:54:35,570 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-16 00:54:35,570 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681606475556, max timestamp=1681606475568Num uniques 859; 2023-04-16 00:54:35,571 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-16 00:54:35,584 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:35,584 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 209 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34718 deadline: 1681606535580, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:35,586 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:35,586 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 207 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34714 deadline: 1681606535583, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:35,586 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:35,586 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 203 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34680 deadline: 1681606535583, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:35,586 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:35,586 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 209 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34702 deadline: 1681606535584, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:35,686 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:35,686 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 211 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34718 deadline: 1681606535685, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:35,688 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:35,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 205 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34680 deadline: 1681606535687, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:35,688 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:35,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 209 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34714 deadline: 1681606535687, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:35,689 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:35,689 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 211 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34702 deadline: 1681606535687, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:35,828 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304167613a856bc244e709162d9d798212feb to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304167613a856bc244e709162d9d798212feb 2023-04-16 00:54:35,892 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:35,892 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 213 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34718 deadline: 1681606535888, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:35,892 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:35,892 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 207 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34680 deadline: 1681606535889, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:35,892 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:35,893 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 211 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34714 deadline: 1681606535889, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:35,894 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:35,894 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 213 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34702 deadline: 1681606535890, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:36,195 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:36,195 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 209 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34680 deadline: 1681606536193, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:36,195 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:36,195 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 215 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34702 deadline: 1681606536194, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:36,195 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:36,195 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 215 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34718 deadline: 1681606536195, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:36,199 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:36,199 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 213 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34714 deadline: 1681606536195, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:36,234 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=452, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/.tmp/A/e4eeebd83aec4bf9a572b652d6aff8ad 2023-04-16 00:54:36,650 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=452 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/.tmp/B/c8d0acf730a7459492179f5740a9235f 2023-04-16 00:54:36,670 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=452 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/.tmp/C/55898da5367340c88a160eadeba65d99 2023-04-16 00:54:36,676 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/.tmp/A/e4eeebd83aec4bf9a572b652d6aff8ad as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/A/e4eeebd83aec4bf9a572b652d6aff8ad 2023-04-16 00:54:36,680 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/A/e4eeebd83aec4bf9a572b652d6aff8ad, entries=150, sequenceid=452, filesize=25.4 K 2023-04-16 00:54:36,680 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/.tmp/B/c8d0acf730a7459492179f5740a9235f as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/B/c8d0acf730a7459492179f5740a9235f 2023-04-16 00:54:36,684 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/B/c8d0acf730a7459492179f5740a9235f, entries=150, sequenceid=452, filesize=11.9 K 2023-04-16 00:54:36,685 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/.tmp/C/55898da5367340c88a160eadeba65d99 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/C/55898da5367340c88a160eadeba65d99 2023-04-16 00:54:36,689 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/C/55898da5367340c88a160eadeba65d99, entries=150, sequenceid=452, filesize=11.9 K 2023-04-16 00:54:36,690 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for bcfff279d7c9f90f5c57aef887110e5f in 1276ms, sequenceid=452, compaction requested=true 2023-04-16 00:54:36,690 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for bcfff279d7c9f90f5c57aef887110e5f: 2023-04-16 00:54:36,690 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f. 2023-04-16 00:54:36,690 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:54:36,690 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:54:36,690 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:36,690 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:54:36,690 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:54:36,693 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:36,693 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:36,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:36,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:36,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:36,693 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:36,693 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:54:36,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:36,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:36,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:36,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:36,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:36,695 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:54:36,695 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3ed78cfa[Count = 0] remaining members to acquire global barrier 2023-04-16 00:54:36,695 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:54:36,695 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:36,696 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:36,696 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:36,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:36,697 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:54:36,697 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:54:36,697 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:54:36,697 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:36,697 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:54:36,698 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:36,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:36,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:36,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:36,698 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:54:36,698 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:54:36,698 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:54:36,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:36,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:36,700 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(9158): Flush requested on bcfff279d7c9f90f5c57aef887110e5f 2023-04-16 00:54:36,700 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing bcfff279d7c9f90f5c57aef887110e5f 3/3 column families, dataSize=140.89 KB heapSize=400.22 KB 2023-04-16 00:54:36,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:36,703 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1681606475575, max timestamp=1681606476700Num uniques 1068; 2023-04-16 00:54:36,703 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=7688, suffix heap size=139040,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=168 2023-04-16 00:54:36,703 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1681606475575, max timestamp=1681606476700Num uniques 1068; 2023-04-16 00:54:36,703 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=9978, suffix heap size=139040,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=218 2023-04-16 00:54:36,703 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bcfff279d7c9f90f5c57aef887110e5f, store=A 2023-04-16 00:54:36,703 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 00:54:36,703 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bcfff279d7c9f90f5c57aef887110e5f, store=B 2023-04-16 00:54:36,703 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 00:54:36,704 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bcfff279d7c9f90f5c57aef887110e5f, store=C 2023-04-16 00:54:36,704 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-16 00:54:36,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:36,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:36,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:36,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:36,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:54:36,716 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:54:36,716 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:54:36,716 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:54:36,716 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:54:36,716 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:54:36,716 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:54:36,718 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:36,719 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:36,719 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:54:36,719 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:36,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:36,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:36,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:36,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:36,719 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:54:36,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:36,719 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:36,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:54:36,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:36,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:36,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:36,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:36,721 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:36,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:36,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:36,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:36,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:36,725 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:36,725 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:36,725 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:54:36,725 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:54:36,725 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:36,725 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:36,725 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:36,725 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:36,725 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:36,725 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:54:36,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:54:36,725 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:54:36,725 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:54:36,725 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:54:36,725 DEBUG [Thread-984] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:54:36,726 DEBUG [Thread-984] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:54:36,726 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:54:36,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:54:36,739 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:36,739 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 222 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34702 deadline: 1681606536737, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:36,739 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:36,739 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 217 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34680 deadline: 1681606536737, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:36,742 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:36,742 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:36,742 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 218 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34714 deadline: 1681606536738, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:36,742 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 222 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34718 deadline: 1681606536739, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:36,750 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230416224b4fc0eba64046ab600d8bf992877f to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230416224b4fc0eba64046ab600d8bf992877f 2023-04-16 00:54:36,826 DEBUG [Thread-984] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:54:36,826 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:54:36,828 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:54:36,829 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:54:36,829 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:54:36,829 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:54:36,829 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:54:36,829 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:54:36,829 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:54:36,830 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:36,830 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:36,831 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:54:36,831 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:36,831 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:54:36,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:54:36,832 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:36,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:36,832 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:54:36,832 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:36,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:54:36,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:36,832 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:54:36,833 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:54:36,833 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:54:36,833 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:54:36,833 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:54:36,833 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:54:36,833 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:54:36,833 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:54:36,833 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f. 2023-04-16 00:54:36,833 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f. started... 2023-04-16 00:54:36,833 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f. as already flushing 2023-04-16 00:54:36,833 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f. 2023-04-16 00:54:36,833 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:54:36,833 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:54:36,833 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:36,833 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:54:36,833 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:54:36,835 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:36,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:36,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:36,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:36,836 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:36,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:36,836 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:36,836 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:54:36,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:36,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:36,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:36,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:36,837 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:54:36,837 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2f432c18[Count = 0] remaining members to acquire global barrier 2023-04-16 00:54:36,837 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:54:36,837 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:36,839 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:36,839 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:36,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:36,839 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:54:36,839 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:54:36,839 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:54:36,839 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:36,839 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:54:36,841 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:36,841 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:54:36,841 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:54:36,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:36,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:36,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:36,841 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:54:36,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:36,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:36,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:36,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:36,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:36,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:36,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:36,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:54:36,844 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:54:36,845 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:54:36,845 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:54:36,845 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:54:36,845 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:54:36,845 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:54:36,846 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:36,846 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:36,846 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 224 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34702 deadline: 1681606536840, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:36,846 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:36,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:36,846 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:54:36,847 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:36,847 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 219 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34680 deadline: 1681606536840, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:36,847 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:36,847 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 224 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34718 deadline: 1681606536843, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:36,847 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:36,847 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:54:36,847 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 220 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34714 deadline: 1681606536843, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:36,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:54:36,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:36,848 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:36,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:36,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:36,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:36,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:36,848 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:36,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:36,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:36,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:36,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:36,850 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:36,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:36,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:36,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:36,852 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:36,852 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:36,852 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:36,852 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:54:36,852 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:54:36,852 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:54:36,852 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:54:36,852 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:36,852 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:36,852 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:36,852 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:36,852 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:54:36,853 DEBUG [Thread-984] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:54:36,853 DEBUG [Thread-984] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:54:36,853 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:54:36,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:54:36,853 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:54:36,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:54:36,953 DEBUG [Thread-984] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:54:36,953 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:54:36,955 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:54:36,956 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:54:36,956 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:54:36,956 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:54:36,956 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:54:36,956 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:54:36,956 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:54:36,956 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:36,956 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:36,958 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:54:36,958 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:36,958 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:54:36,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:54:36,958 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:36,958 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:54:36,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:36,959 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:36,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:54:36,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:36,959 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:54:36,959 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:54:36,959 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:54:36,959 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:54:36,959 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:54:36,959 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:54:36,960 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:54:36,960 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:54:36,960 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f. 2023-04-16 00:54:36,960 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f. started... 2023-04-16 00:54:36,960 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f. as already flushing 2023-04-16 00:54:36,960 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f. 2023-04-16 00:54:36,960 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:54:36,960 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:54:36,960 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:36,960 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:54:36,960 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:54:36,962 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:36,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:36,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:36,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:36,963 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:36,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:36,963 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:36,963 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:54:36,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:36,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:36,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:36,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:36,964 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:54:36,964 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:54:36,964 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3f22fd03[Count = 0] remaining members to acquire global barrier 2023-04-16 00:54:36,964 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:36,966 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:36,966 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:36,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:36,966 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:54:36,966 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:54:36,966 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:36,966 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:54:36,966 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:54:36,967 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:36,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:36,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:36,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:36,967 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:54:36,967 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:54:36,967 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:54:36,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:36,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:36,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:36,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:36,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:36,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:36,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:36,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:54:36,969 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:54:36,969 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:54:36,969 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:54:36,969 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:54:36,969 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:54:36,969 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:54:36,971 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:36,971 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:36,971 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:54:36,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:36,971 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:36,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:36,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:36,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:36,971 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:54:36,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:54:36,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:36,972 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:36,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:36,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:36,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:36,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:36,973 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:36,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:36,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:36,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:36,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:36,978 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:36,978 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:36,978 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:36,978 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:36,978 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:54:36,978 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:54:36,979 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:54:36,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:54:36,979 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:54:36,978 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:54:36,978 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:54:36,979 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:54:36,979 DEBUG [Thread-984] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:54:36,979 DEBUG [Thread-984] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:54:36,978 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:36,979 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:54:36,979 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:36,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:54:36,979 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:37,048 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:37,048 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 221 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34680 deadline: 1681606537048, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:37,051 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:37,051 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 226 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34702 deadline: 1681606537048, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:37,051 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:37,051 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 226 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34718 deadline: 1681606537049, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:37,051 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:37,051 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 222 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34714 deadline: 1681606537049, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:37,079 DEBUG [Thread-984] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:54:37,080 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:54:37,081 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:54:37,082 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:54:37,082 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:54:37,082 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:54:37,082 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:54:37,082 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:54:37,083 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:54:37,083 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:37,083 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:37,084 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:54:37,084 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:37,084 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:54:37,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:54:37,085 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:37,085 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:54:37,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:37,085 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:37,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:54:37,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:37,085 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:54:37,085 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:54:37,086 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:54:37,086 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:54:37,086 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:54:37,086 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:54:37,086 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:54:37,086 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:54:37,086 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f. 2023-04-16 00:54:37,086 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f. started... 2023-04-16 00:54:37,086 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f. as already flushing 2023-04-16 00:54:37,086 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f. 2023-04-16 00:54:37,086 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:54:37,086 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:54:37,086 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:37,086 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:54:37,086 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:54:37,088 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:37,088 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:37,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:37,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:37,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:37,088 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:37,089 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:54:37,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:37,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:37,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:37,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:37,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:37,090 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:54:37,090 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:54:37,090 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:37,090 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2d787516[Count = 0] remaining members to acquire global barrier 2023-04-16 00:54:37,092 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:37,092 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:37,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:37,092 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:54:37,092 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:54:37,092 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:37,092 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:54:37,092 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:54:37,095 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:37,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:37,095 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:54:37,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:37,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:37,095 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:54:37,095 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:54:37,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:37,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:37,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:37,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:37,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:37,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:37,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:37,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:54:37,098 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:54:37,098 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:54:37,099 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:54:37,099 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:54:37,099 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:54:37,099 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:54:37,100 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:37,100 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:37,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:37,100 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:37,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:37,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:37,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:37,100 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:54:37,101 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:54:37,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:54:37,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:37,101 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:37,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:37,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:37,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:37,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:37,103 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:37,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:37,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:37,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:37,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:37,106 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:54:37,106 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:37,106 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:54:37,106 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:54:37,106 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:54:37,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:54:37,106 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:54:37,106 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:54:37,106 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:37,106 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:54:37,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:54:37,106 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:37,106 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:37,106 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:37,106 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:37,107 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:37,106 DEBUG [Thread-984] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:54:37,107 DEBUG [Thread-984] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:54:37,160 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=490, memsize=11.2 K, hasBloomFilter=true, into tmp file hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/.tmp/A/43842ce68c6f4ce0beee6ab316623391 2023-04-16 00:54:37,207 DEBUG [Thread-984] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:54:37,207 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:54:37,209 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:54:37,210 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:54:37,210 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:54:37,210 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:54:37,210 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:54:37,210 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:54:37,210 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:54:37,211 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:37,211 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:37,212 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:54:37,212 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:37,212 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:54:37,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:54:37,212 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:37,212 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:54:37,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:37,213 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:37,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:54:37,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:37,213 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:54:37,213 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:54:37,213 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:54:37,213 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:54:37,213 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:54:37,213 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:54:37,214 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:54:37,214 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:54:37,214 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f. 2023-04-16 00:54:37,214 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f. started... 2023-04-16 00:54:37,214 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f. as already flushing 2023-04-16 00:54:37,214 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f. 2023-04-16 00:54:37,214 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:54:37,215 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:54:37,215 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:37,215 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:54:37,215 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:54:37,217 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:37,217 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:37,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:37,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:37,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:37,218 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:37,218 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:54:37,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:37,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:37,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:37,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:37,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:37,219 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:54:37,219 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:54:37,219 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2fca8c1f[Count = 0] remaining members to acquire global barrier 2023-04-16 00:54:37,219 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:37,220 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:37,220 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:37,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:37,220 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:54:37,220 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:37,220 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:54:37,220 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:54:37,221 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:54:37,222 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:54:37,222 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:54:37,222 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:37,222 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:54:37,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:37,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:37,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:37,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:37,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:37,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:37,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:37,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:37,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:37,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:37,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:54:37,224 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:54:37,224 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:54:37,224 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:54:37,224 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:54:37,224 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:54:37,224 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:54:37,226 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:37,226 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:54:37,226 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:37,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:37,226 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:37,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:37,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:37,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:37,226 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:54:37,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:54:37,227 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:37,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:37,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:37,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:37,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:37,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:37,228 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:37,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:37,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:37,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:37,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:37,231 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:37,231 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:37,231 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:54:37,231 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:37,231 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:54:37,231 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:37,231 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:37,231 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:37,231 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:37,231 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:54:37,231 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:54:37,231 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:54:37,231 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:54:37,231 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:54:37,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:54:37,231 DEBUG [Thread-984] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:54:37,231 DEBUG [Thread-984] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:54:37,232 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:54:37,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:54:37,332 DEBUG [Thread-984] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:54:37,332 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:54:37,333 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:54:37,334 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:54:37,334 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:54:37,334 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:54:37,334 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:54:37,334 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:54:37,334 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:54:37,335 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:37,335 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:37,336 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:54:37,336 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:37,336 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:54:37,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:54:37,336 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:37,336 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:54:37,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:37,337 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:37,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:54:37,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:37,337 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:54:37,337 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:54:37,338 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:54:37,338 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:54:37,338 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:54:37,338 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:54:37,338 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:54:37,338 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:54:37,338 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f. 2023-04-16 00:54:37,338 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f. started... 2023-04-16 00:54:37,338 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f. as already flushing 2023-04-16 00:54:37,338 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f. 2023-04-16 00:54:37,338 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:54:37,338 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:54:37,338 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:37,338 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:54:37,338 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:54:37,340 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:37,340 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:37,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:37,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:37,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:37,340 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:37,340 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:54:37,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:37,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:37,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:37,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:37,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:37,342 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:54:37,342 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@19df5612[Count = 0] remaining members to acquire global barrier 2023-04-16 00:54:37,342 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:54:37,342 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:37,343 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:37,343 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:37,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:37,343 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:54:37,343 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:54:37,343 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:54:37,343 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:37,343 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:54:37,345 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:37,345 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:54:37,345 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:54:37,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:37,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:37,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:37,345 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:54:37,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:37,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:37,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:37,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:37,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:37,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:37,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:37,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:54:37,348 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:54:37,348 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:54:37,348 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:54:37,348 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:54:37,348 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:54:37,348 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:54:37,349 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:37,349 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:37,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:37,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:37,350 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:54:37,350 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:37,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:37,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:37,350 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:54:37,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:54:37,350 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:37,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:37,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:37,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:37,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:37,352 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:37,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:37,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:37,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:37,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:37,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:37,353 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:37,353 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 223 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34680 deadline: 1681606537351, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:37,353 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:37,354 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 228 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34702 deadline: 1681606537352, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:37,354 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:37,354 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 228 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34718 deadline: 1681606537352, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:37,354 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:37,354 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 224 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34714 deadline: 1681606537353, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:37,357 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:37,357 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:37,357 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:37,357 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:54:37,357 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:54:37,357 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:37,357 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:54:37,357 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:54:37,357 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:54:37,357 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:54:37,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:54:37,357 DEBUG [Thread-984] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:54:37,357 DEBUG [Thread-984] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:54:37,357 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:37,357 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:37,357 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:37,358 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:54:37,358 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:54:37,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:54:37,458 DEBUG [Thread-984] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:54:37,458 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:54:37,459 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:54:37,460 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:54:37,460 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:54:37,460 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:54:37,460 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:54:37,461 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:54:37,461 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:54:37,461 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:37,461 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:37,463 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:54:37,463 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:37,463 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:54:37,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:54:37,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:37,463 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:37,463 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:54:37,464 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:37,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:54:37,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:37,464 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:54:37,464 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:54:37,464 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:54:37,464 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:54:37,464 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:54:37,464 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:54:37,465 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:54:37,465 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:54:37,465 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f. 2023-04-16 00:54:37,465 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f. started... 2023-04-16 00:54:37,465 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f. as already flushing 2023-04-16 00:54:37,465 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f. 2023-04-16 00:54:37,465 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:54:37,465 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:54:37,465 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:37,465 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:54:37,465 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:54:37,466 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:37,466 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:37,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:37,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:37,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:37,467 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:37,467 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:54:37,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:37,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:37,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:37,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:37,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:37,468 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:54:37,468 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:54:37,468 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@313d6acf[Count = 0] remaining members to acquire global barrier 2023-04-16 00:54:37,468 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:37,470 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:37,470 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:37,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:37,470 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:54:37,470 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:37,470 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:54:37,470 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:54:37,470 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:54:37,471 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:37,471 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:54:37,471 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:54:37,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:37,472 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:54:37,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:37,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:37,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:37,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:37,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:37,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:37,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:37,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:37,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:37,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:54:37,474 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:54:37,474 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:54:37,474 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:54:37,474 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:54:37,474 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:54:37,474 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:54:37,476 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:37,476 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:37,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:37,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:37,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:37,476 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:37,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:37,476 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:54:37,477 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:54:37,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:54:37,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:37,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:37,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:37,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:37,478 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:37,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:37,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:37,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:37,479 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:37,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:37,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:37,481 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:54:37,481 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:37,481 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:54:37,481 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:37,481 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:54:37,482 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:54:37,482 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:37,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:54:37,482 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:37,482 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:54:37,482 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:54:37,482 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:37,482 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:37,482 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:37,482 DEBUG [Thread-984] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:54:37,482 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:54:37,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:54:37,482 DEBUG [Thread-984] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:54:37,554 DEBUG [Thread-989] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3f43b712 to 127.0.0.1:53150 2023-04-16 00:54:37,554 DEBUG [Thread-989] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-16 00:54:37,556 DEBUG [Thread-987] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2c4379c2 to 127.0.0.1:53150 2023-04-16 00:54:37,556 DEBUG [Thread-987] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-16 00:54:37,557 DEBUG [Thread-985] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x54451f11 to 127.0.0.1:53150 2023-04-16 00:54:37,557 DEBUG [Thread-985] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-16 00:54:37,558 DEBUG [Thread-991] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7a3bb940 to 127.0.0.1:53150 2023-04-16 00:54:37,558 DEBUG [Thread-991] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-16 00:54:37,571 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=490 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/.tmp/B/ae2b7c794b4a44f09914e452821e7dd2 2023-04-16 00:54:37,582 DEBUG [Thread-984] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:54:37,583 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:54:37,855 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:37,855 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:37,855 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 225 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34680 deadline: 1681606537854, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:37,855 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 230 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34702 deadline: 1681606537854, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:37,858 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:37,858 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:37,858 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 230 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34718 deadline: 1681606537858, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:37,858 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 226 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34714 deadline: 1681606537858, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:37,980 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=490 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/.tmp/C/f60febfbc08f4af5b4af6be1c8720f06 2023-04-16 00:54:37,983 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/.tmp/A/43842ce68c6f4ce0beee6ab316623391 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/A/43842ce68c6f4ce0beee6ab316623391 2023-04-16 00:54:37,986 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/A/43842ce68c6f4ce0beee6ab316623391, entries=200, sequenceid=490, filesize=32.3 K 2023-04-16 00:54:37,987 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/.tmp/B/ae2b7c794b4a44f09914e452821e7dd2 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/B/ae2b7c794b4a44f09914e452821e7dd2 2023-04-16 00:54:37,990 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/B/ae2b7c794b4a44f09914e452821e7dd2, entries=150, sequenceid=490, filesize=11.9 K 2023-04-16 00:54:37,991 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/.tmp/C/f60febfbc08f4af5b4af6be1c8720f06 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/C/f60febfbc08f4af5b4af6be1c8720f06 2023-04-16 00:54:37,994 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/C/f60febfbc08f4af5b4af6be1c8720f06, entries=150, sequenceid=490, filesize=11.9 K 2023-04-16 00:54:37,995 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~69.33 KB/70990, heapSize ~188.91 KB/193440, currentSize=114.05 KB/116790 for bcfff279d7c9f90f5c57aef887110e5f in 1295ms, sequenceid=490, compaction requested=true 2023-04-16 00:54:37,995 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for bcfff279d7c9f90f5c57aef887110e5f: 2023-04-16 00:54:37,995 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 00:54:37,995 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-16 00:54:37,995 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-16 00:54:37,995 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 00:54:37,996 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 00:54:37,997 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 112008 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-16 00:54:37,997 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HStore(1912): bcfff279d7c9f90f5c57aef887110e5f/A is initiating minor compaction (all files) 2023-04-16 00:54:37,997 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49528 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-16 00:54:37,997 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of bcfff279d7c9f90f5c57aef887110e5f/A in TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f. 2023-04-16 00:54:37,997 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HStore(1912): bcfff279d7c9f90f5c57aef887110e5f/B is initiating minor compaction (all files) 2023-04-16 00:54:37,997 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/A/0eaeed5ba5f34552934498a46e2aef33, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/A/664e173ad4404213bab38d250b20769c, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/A/e4eeebd83aec4bf9a572b652d6aff8ad, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/A/43842ce68c6f4ce0beee6ab316623391] into tmpdir=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/.tmp, totalSize=109.4 K 2023-04-16 00:54:37,997 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HRegion(2259): Starting compaction of bcfff279d7c9f90f5c57aef887110e5f/B in TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f. 2023-04-16 00:54:37,997 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/B/d921f30e70c84bed960a37d28b43b8c4, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/B/40c2c459df1a4c26b6277e187e31a9cc, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/B/c8d0acf730a7459492179f5740a9235f, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/B/ae2b7c794b4a44f09914e452821e7dd2] into tmpdir=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/.tmp, totalSize=48.4 K 2023-04-16 00:54:37,998 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.Compactor(207): Compacting 0eaeed5ba5f34552934498a46e2aef33, keycount=150, bloomtype=ROW, size=26.2 K, encoding=NONE, compression=NONE, seqNum=404, earliestPutTs=1681606473749 2023-04-16 00:54:37,998 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting d921f30e70c84bed960a37d28b43b8c4, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=404, earliestPutTs=1681606473749 2023-04-16 00:54:37,998 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.Compactor(207): Compacting 664e173ad4404213bab38d250b20769c, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=440, earliestPutTs=1681606473802 2023-04-16 00:54:37,998 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting 40c2c459df1a4c26b6277e187e31a9cc, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=440, earliestPutTs=1681606473802 2023-04-16 00:54:37,998 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.Compactor(207): Compacting e4eeebd83aec4bf9a572b652d6aff8ad, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=452, earliestPutTs=1681606474429 2023-04-16 00:54:37,999 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting c8d0acf730a7459492179f5740a9235f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=452, earliestPutTs=1681606474429 2023-04-16 00:54:37,999 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.Compactor(207): Compacting 43842ce68c6f4ce0beee6ab316623391, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=490, earliestPutTs=1681606475575 2023-04-16 00:54:37,999 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting ae2b7c794b4a44f09914e452821e7dd2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=490, earliestPutTs=1681606475575 2023-04-16 00:54:38,008 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] throttle.PressureAwareThroughputController(145): bcfff279d7c9f90f5c57aef887110e5f#B#compaction#208 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-16 00:54:38,022 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] throttle.PressureAwareThroughputController(145): bcfff279d7c9f90f5c57aef887110e5f#A#compaction#207 average throughput is 1.23 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-16 00:54:38,426 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/.tmp/B/1bfa6c62814b4968ba4d40f18210fc55 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/B/1bfa6c62814b4968ba4d40f18210fc55 2023-04-16 00:54:38,430 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in bcfff279d7c9f90f5c57aef887110e5f/B of bcfff279d7c9f90f5c57aef887110e5f into 1bfa6c62814b4968ba4d40f18210fc55(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 00:54:38,430 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for bcfff279d7c9f90f5c57aef887110e5f: 2023-04-16 00:54:38,430 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f., storeName=bcfff279d7c9f90f5c57aef887110e5f/B, priority=12, startTime=1681606477995; duration=0sec 2023-04-16 00:54:38,431 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 00:54:38,431 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-16 00:54:38,432 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49528 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-16 00:54:38,432 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HStore(1912): bcfff279d7c9f90f5c57aef887110e5f/C is initiating minor compaction (all files) 2023-04-16 00:54:38,432 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HRegion(2259): Starting compaction of bcfff279d7c9f90f5c57aef887110e5f/C in TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f. 2023-04-16 00:54:38,432 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/C/68c6722fdd444523ac341216c11dcdc4, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/C/0f00134bae7c4109a12dd915c06cdbc0, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/C/55898da5367340c88a160eadeba65d99, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/C/f60febfbc08f4af5b4af6be1c8720f06] into tmpdir=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/.tmp, totalSize=48.4 K 2023-04-16 00:54:38,433 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting 68c6722fdd444523ac341216c11dcdc4, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=404, earliestPutTs=1681606473749 2023-04-16 00:54:38,433 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting 0f00134bae7c4109a12dd915c06cdbc0, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=440, earliestPutTs=1681606473802 2023-04-16 00:54:38,433 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting 55898da5367340c88a160eadeba65d99, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=452, earliestPutTs=1681606474429 2023-04-16 00:54:38,434 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting f60febfbc08f4af5b4af6be1c8720f06, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=490, earliestPutTs=1681606475575 2023-04-16 00:54:38,445 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] throttle.PressureAwareThroughputController(145): bcfff279d7c9f90f5c57aef887110e5f#C#compaction#209 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-16 00:54:38,449 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/.tmp/A/051d11b1c1004549ba4b17546e17c613 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/A/051d11b1c1004549ba4b17546e17c613 2023-04-16 00:54:38,453 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/.tmp/C/a86fe8878e3d4cd6aa412f7eae479c31 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/C/a86fe8878e3d4cd6aa412f7eae479c31 2023-04-16 00:54:38,454 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in bcfff279d7c9f90f5c57aef887110e5f/A of bcfff279d7c9f90f5c57aef887110e5f into 051d11b1c1004549ba4b17546e17c613(size=26.3 K), total size for store is 26.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 00:54:38,454 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for bcfff279d7c9f90f5c57aef887110e5f: 2023-04-16 00:54:38,454 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f., storeName=bcfff279d7c9f90f5c57aef887110e5f/A, priority=12, startTime=1681606477995; duration=0sec 2023-04-16 00:54:38,454 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 00:54:38,457 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in bcfff279d7c9f90f5c57aef887110e5f/C of bcfff279d7c9f90f5c57aef887110e5f into a86fe8878e3d4cd6aa412f7eae479c31(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 00:54:38,457 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for bcfff279d7c9f90f5c57aef887110e5f: 2023-04-16 00:54:38,457 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f., storeName=bcfff279d7c9f90f5c57aef887110e5f/C, priority=12, startTime=1681606477996; duration=0sec 2023-04-16 00:54:38,457 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 00:54:38,857 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(9158): Flush requested on bcfff279d7c9f90f5c57aef887110e5f 2023-04-16 00:54:38,857 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing bcfff279d7c9f90f5c57aef887110e5f 3/3 column families, dataSize=120.76 KB heapSize=360.05 KB 2023-04-16 00:54:38,857 DEBUG [Thread-980] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x69f15cb4 to 127.0.0.1:53150 2023-04-16 00:54:38,857 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bcfff279d7c9f90f5c57aef887110e5f, store=A 2023-04-16 00:54:38,857 DEBUG [Thread-980] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-16 00:54:38,857 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 00:54:38,857 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bcfff279d7c9f90f5c57aef887110e5f, store=B 2023-04-16 00:54:38,858 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 00:54:38,858 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bcfff279d7c9f90f5c57aef887110e5f, store=C 2023-04-16 00:54:38,858 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 00:54:38,861 DEBUG [Thread-982] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6c1c92c6 to 127.0.0.1:53150 2023-04-16 00:54:38,861 DEBUG [Thread-982] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-16 00:54:38,866 DEBUG [Thread-976] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1aa5eef5 to 127.0.0.1:53150 2023-04-16 00:54:38,866 DEBUG [Thread-974] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5b3934fb to 127.0.0.1:53150 2023-04-16 00:54:38,866 DEBUG [Thread-974] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-16 00:54:38,866 DEBUG [Thread-976] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-16 00:54:39,270 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304162f4daf5a859d4f1d8c9b6babea9cfc01 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304162f4daf5a859d4f1d8c9b6babea9cfc01 2023-04-16 00:54:39,675 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=514, memsize=40.3 K, hasBloomFilter=true, into tmp file hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/.tmp/A/0b2360c43fd64951ab6f66ac174d825f 2023-04-16 00:54:40,084 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=514 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/.tmp/B/cd7f6fe0d360478ab00ecd5d819094f2 2023-04-16 00:54:40,493 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=514 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/.tmp/C/03355f8c3d4f4994bc4850a253c0f75e 2023-04-16 00:54:40,498 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/.tmp/A/0b2360c43fd64951ab6f66ac174d825f as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/A/0b2360c43fd64951ab6f66ac174d825f 2023-04-16 00:54:40,501 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/A/0b2360c43fd64951ab6f66ac174d825f, entries=150, sequenceid=514, filesize=25.4 K 2023-04-16 00:54:40,501 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/.tmp/B/cd7f6fe0d360478ab00ecd5d819094f2 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/B/cd7f6fe0d360478ab00ecd5d819094f2 2023-04-16 00:54:40,505 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/B/cd7f6fe0d360478ab00ecd5d819094f2, entries=150, sequenceid=514, filesize=11.9 K 2023-04-16 00:54:40,505 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/.tmp/C/03355f8c3d4f4994bc4850a253c0f75e as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/C/03355f8c3d4f4994bc4850a253c0f75e 2023-04-16 00:54:40,509 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/C/03355f8c3d4f4994bc4850a253c0f75e, entries=150, sequenceid=514, filesize=11.9 K 2023-04-16 00:54:40,510 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~120.76 KB/123660, heapSize ~360 KB/368640, currentSize=20.13 KB/20610 for bcfff279d7c9f90f5c57aef887110e5f in 1653ms, sequenceid=514, compaction requested=false 2023-04-16 00:54:40,510 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for bcfff279d7c9f90f5c57aef887110e5f: 2023-04-16 00:54:40,516 DEBUG [regionserver/jenkins-hbase4:0.Chore.1] throttle.PressureAwareCompactionThroughputController(103): CompactionPressure is 0.0, tune throughput to 50.00 MB/second 2023-04-16 00:54:41,162 DEBUG [Thread-978] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1441c9f0 to 127.0.0.1:53150 2023-04-16 00:54:41,162 DEBUG [Thread-978] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-16 00:54:41,162 INFO [Listener at localhost/33477] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-16 00:54:41,162 INFO [Listener at localhost/33477] hbase.AcidGuaranteesTestTool(394): wrote 93 2023-04-16 00:54:41,163 INFO [Listener at localhost/33477] hbase.AcidGuaranteesTestTool(394): wrote 103 2023-04-16 00:54:41,163 INFO [Listener at localhost/33477] hbase.AcidGuaranteesTestTool(394): wrote 33 2023-04-16 00:54:41,163 INFO [Listener at localhost/33477] hbase.AcidGuaranteesTestTool(394): wrote 92 2023-04-16 00:54:41,163 INFO [Listener at localhost/33477] hbase.AcidGuaranteesTestTool(394): wrote 97 2023-04-16 00:54:41,163 INFO [Listener at localhost/33477] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-16 00:54:41,163 INFO [Listener at localhost/33477] hbase.AcidGuaranteesTestTool(398): read 4646 2023-04-16 00:54:41,163 INFO [Listener at localhost/33477] hbase.AcidGuaranteesTestTool(398): read 4626 2023-04-16 00:54:41,163 INFO [Listener at localhost/33477] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-16 00:54:41,163 INFO [Listener at localhost/33477] hbase.AcidGuaranteesTestTool(402): scanned 1976 2023-04-16 00:54:41,163 INFO [Listener at localhost/33477] hbase.AcidGuaranteesTestTool(403): verified 5927 rows 2023-04-16 00:54:41,163 INFO [Listener at localhost/33477] hbase.AcidGuaranteesTestTool(402): scanned 1989 2023-04-16 00:54:41,163 INFO [Listener at localhost/33477] hbase.AcidGuaranteesTestTool(403): verified 5967 rows 2023-04-16 00:54:41,163 INFO [Listener at localhost/33477] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-16 00:54:41,163 DEBUG [Listener at localhost/33477] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x33c943f0 to 127.0.0.1:53150 2023-04-16 00:54:41,163 DEBUG [Listener at localhost/33477] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-16 00:54:41,165 INFO [Listener at localhost/33477] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-16 00:54:41,165 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable TestAcidGuarantees 2023-04-16 00:54:41,166 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure2.ProcedureExecutor(1029): Stored pid=24, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-16 00:54:41,170 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-16 00:54:41,170 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681606481170"}]},"ts":"1681606481170"} 2023-04-16 00:54:41,171 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-16 00:54:41,173 INFO [PEWorker-5] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-16 00:54:41,173 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=25, ppid=24, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=bcfff279d7c9f90f5c57aef887110e5f, UNASSIGN}] 2023-04-16 00:54:41,175 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=25, ppid=24, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=bcfff279d7c9f90f5c57aef887110e5f, UNASSIGN 2023-04-16 00:54:41,175 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=25 updating hbase:meta row=bcfff279d7c9f90f5c57aef887110e5f, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:41,175 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681606481175"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681606481175"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681606481175"}]},"ts":"1681606481175"} 2023-04-16 00:54:41,176 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=26, ppid=25, state=RUNNABLE; CloseRegionProcedure bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217}] 2023-04-16 00:54:41,270 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-16 00:54:41,328 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close bcfff279d7c9f90f5c57aef887110e5f 2023-04-16 00:54:41,328 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing bcfff279d7c9f90f5c57aef887110e5f, disabling compactions & flushes 2023-04-16 00:54:41,328 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f. 2023-04-16 00:54:41,328 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f. 2023-04-16 00:54:41,328 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f. after waiting 0 ms 2023-04-16 00:54:41,328 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f. 2023-04-16 00:54:41,328 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing bcfff279d7c9f90f5c57aef887110e5f 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-04-16 00:54:41,328 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bcfff279d7c9f90f5c57aef887110e5f, store=A 2023-04-16 00:54:41,328 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:54:41,328 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bcfff279d7c9f90f5c57aef887110e5f, store=B 2023-04-16 00:54:41,328 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:54:41,328 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bcfff279d7c9f90f5c57aef887110e5f, store=C 2023-04-16 00:54:41,328 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:54:41,471 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-16 00:54:41,741 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230416ef9160b9b3b740cbb675ced437274422 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230416ef9160b9b3b740cbb675ced437274422 2023-04-16 00:54:41,772 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-16 00:54:42,146 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=521, memsize=8.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/.tmp/A/8d86a17a3e8c49cdb362ff8c36df9f9b 2023-04-16 00:54:42,273 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-16 00:54:42,554 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=521 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/.tmp/B/11ed392fa0f9417885e48be6071f9676 2023-04-16 00:54:42,963 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=521 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/.tmp/C/0c61ec02d47746798df819782d4126a4 2023-04-16 00:54:42,967 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/.tmp/A/8d86a17a3e8c49cdb362ff8c36df9f9b as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/A/8d86a17a3e8c49cdb362ff8c36df9f9b 2023-04-16 00:54:42,970 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/A/8d86a17a3e8c49cdb362ff8c36df9f9b, entries=150, sequenceid=521, filesize=25.4 K 2023-04-16 00:54:42,970 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/.tmp/B/11ed392fa0f9417885e48be6071f9676 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/B/11ed392fa0f9417885e48be6071f9676 2023-04-16 00:54:42,973 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/B/11ed392fa0f9417885e48be6071f9676, entries=150, sequenceid=521, filesize=11.9 K 2023-04-16 00:54:42,974 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/.tmp/C/0c61ec02d47746798df819782d4126a4 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/C/0c61ec02d47746798df819782d4126a4 2023-04-16 00:54:42,977 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/C/0c61ec02d47746798df819782d4126a4, entries=150, sequenceid=521, filesize=11.9 K 2023-04-16 00:54:42,977 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=0 B/0 for bcfff279d7c9f90f5c57aef887110e5f in 1649ms, sequenceid=521, compaction requested=true 2023-04-16 00:54:42,997 DEBUG [StoreCloser-TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/A/e1d6985772b2459891a09ef7b6f7a71a, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/A/c3bc52b00b004f1ead6c43393355b4f9, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/A/d42d49a6ea17498eafcd86ca67f9d886, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/A/d94226ccd6774623b5d4f922c8e69e9e, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/A/562893653eb54f86ac7befc2f34095cc, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/A/323f8a53904c4a2f8e2f130ee9dd9392, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/A/ccfd106fb798442d87f12fc11d53a8e5, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/A/7ec2d64efa9d42c88357a6e4c185ee0c, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/A/5d3dcad2480e49d486ddb3a48152bdf2, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/A/430045034cd042ee8842b99fce78fad7, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/A/7e589a35f7264ea482484f88a05d763c, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/A/26621a1cb7934592b7abc6dc6df9a986, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/A/075edb7eb8c447e989647b62da937d3d, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/A/4ccc3b61512544deb085ea689a2f6d38, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/A/fe13155d833d4c6984e41d4f025dcc10, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/A/a6893ea908c745e097b7ab68ba2b400a, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/A/d4a18a42bce44d68a6de859d66c68a1c, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/A/c274a82a5dfb44d3a9d5d0f8b55f9678, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/A/294c519045464f2ebd5223681bbdaa69, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/A/0be1677bce94454eb2808230b3f443de, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/A/4824237334444d00b911471b2a96ef53, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/A/12a77a02ea8c4ca39a9ee6fd692f7bf5, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/A/e0b09a14eb324581b9ca2ec144e1a33e, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/A/0eaeed5ba5f34552934498a46e2aef33, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/A/664e173ad4404213bab38d250b20769c, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/A/e4eeebd83aec4bf9a572b652d6aff8ad, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/A/43842ce68c6f4ce0beee6ab316623391] to archive 2023-04-16 00:54:42,998 DEBUG [StoreCloser-TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-16 00:54:43,000 DEBUG [StoreCloser-TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/A/e1d6985772b2459891a09ef7b6f7a71a to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/A/e1d6985772b2459891a09ef7b6f7a71a 2023-04-16 00:54:43,001 DEBUG [StoreCloser-TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/A/c3bc52b00b004f1ead6c43393355b4f9 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/A/c3bc52b00b004f1ead6c43393355b4f9 2023-04-16 00:54:43,002 DEBUG [StoreCloser-TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/A/d42d49a6ea17498eafcd86ca67f9d886 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/A/d42d49a6ea17498eafcd86ca67f9d886 2023-04-16 00:54:43,003 DEBUG [StoreCloser-TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/A/d94226ccd6774623b5d4f922c8e69e9e to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/A/d94226ccd6774623b5d4f922c8e69e9e 2023-04-16 00:54:43,004 DEBUG [StoreCloser-TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/A/562893653eb54f86ac7befc2f34095cc to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/A/562893653eb54f86ac7befc2f34095cc 2023-04-16 00:54:43,005 DEBUG [StoreCloser-TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/A/323f8a53904c4a2f8e2f130ee9dd9392 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/A/323f8a53904c4a2f8e2f130ee9dd9392 2023-04-16 00:54:43,006 DEBUG [StoreCloser-TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/A/ccfd106fb798442d87f12fc11d53a8e5 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/A/ccfd106fb798442d87f12fc11d53a8e5 2023-04-16 00:54:43,006 DEBUG [StoreCloser-TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/A/7ec2d64efa9d42c88357a6e4c185ee0c to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/A/7ec2d64efa9d42c88357a6e4c185ee0c 2023-04-16 00:54:43,008 DEBUG [StoreCloser-TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/A/5d3dcad2480e49d486ddb3a48152bdf2 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/A/5d3dcad2480e49d486ddb3a48152bdf2 2023-04-16 00:54:43,009 DEBUG [StoreCloser-TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/A/430045034cd042ee8842b99fce78fad7 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/A/430045034cd042ee8842b99fce78fad7 2023-04-16 00:54:43,009 DEBUG [StoreCloser-TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/A/7e589a35f7264ea482484f88a05d763c to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/A/7e589a35f7264ea482484f88a05d763c 2023-04-16 00:54:43,010 DEBUG [StoreCloser-TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/A/26621a1cb7934592b7abc6dc6df9a986 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/A/26621a1cb7934592b7abc6dc6df9a986 2023-04-16 00:54:43,011 DEBUG [StoreCloser-TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/A/075edb7eb8c447e989647b62da937d3d to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/A/075edb7eb8c447e989647b62da937d3d 2023-04-16 00:54:43,012 DEBUG [StoreCloser-TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/A/4ccc3b61512544deb085ea689a2f6d38 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/A/4ccc3b61512544deb085ea689a2f6d38 2023-04-16 00:54:43,013 DEBUG [StoreCloser-TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/A/fe13155d833d4c6984e41d4f025dcc10 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/A/fe13155d833d4c6984e41d4f025dcc10 2023-04-16 00:54:43,014 DEBUG [StoreCloser-TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/A/a6893ea908c745e097b7ab68ba2b400a to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/A/a6893ea908c745e097b7ab68ba2b400a 2023-04-16 00:54:43,015 DEBUG [StoreCloser-TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/A/d4a18a42bce44d68a6de859d66c68a1c to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/A/d4a18a42bce44d68a6de859d66c68a1c 2023-04-16 00:54:43,016 DEBUG [StoreCloser-TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/A/c274a82a5dfb44d3a9d5d0f8b55f9678 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/A/c274a82a5dfb44d3a9d5d0f8b55f9678 2023-04-16 00:54:43,017 DEBUG [StoreCloser-TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/A/294c519045464f2ebd5223681bbdaa69 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/A/294c519045464f2ebd5223681bbdaa69 2023-04-16 00:54:43,018 DEBUG [StoreCloser-TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/A/0be1677bce94454eb2808230b3f443de to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/A/0be1677bce94454eb2808230b3f443de 2023-04-16 00:54:43,018 DEBUG [StoreCloser-TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/A/4824237334444d00b911471b2a96ef53 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/A/4824237334444d00b911471b2a96ef53 2023-04-16 00:54:43,019 DEBUG [StoreCloser-TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/A/12a77a02ea8c4ca39a9ee6fd692f7bf5 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/A/12a77a02ea8c4ca39a9ee6fd692f7bf5 2023-04-16 00:54:43,020 DEBUG [StoreCloser-TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/A/e0b09a14eb324581b9ca2ec144e1a33e to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/A/e0b09a14eb324581b9ca2ec144e1a33e 2023-04-16 00:54:43,021 DEBUG [StoreCloser-TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/A/0eaeed5ba5f34552934498a46e2aef33 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/A/0eaeed5ba5f34552934498a46e2aef33 2023-04-16 00:54:43,022 DEBUG [StoreCloser-TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/A/664e173ad4404213bab38d250b20769c to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/A/664e173ad4404213bab38d250b20769c 2023-04-16 00:54:43,023 DEBUG [StoreCloser-TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/A/e4eeebd83aec4bf9a572b652d6aff8ad to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/A/e4eeebd83aec4bf9a572b652d6aff8ad 2023-04-16 00:54:43,023 DEBUG [StoreCloser-TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/A/43842ce68c6f4ce0beee6ab316623391 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/A/43842ce68c6f4ce0beee6ab316623391 2023-04-16 00:54:43,026 DEBUG [StoreCloser-TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/B/3f116ca5aea14ee089ffd2938d1a6e36, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/B/004575861f934c2da4ab487a312b7e5d, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/B/9d8eae1b34a04e2f9b603066470cb6ca, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/B/a54d08fb83414d0f9473426c7209b4a4, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/B/60b85be4841644a385e443b5d1e20625, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/B/4fe52a20c50b4913b5f13dd3ac7e3bb4, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/B/8e66594f3e484d3eab1c157a1b7e145d, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/B/84950f4f4c394a79867b48b57fbfbb7a, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/B/f02f9a14b5bb4aacaad5dc979ad36280, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/B/05ae3848c7b54920b331066dbd534638, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/B/92aca36b949c48efbb8ad2d0834d0869, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/B/0a265fceaa9f42b5a2112160556b7e93, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/B/f8ed64d1e7e040a78b5f518f4babbe79, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/B/cc2a6584442c4dada7e4049ae45d3ed9, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/B/d6ae0991afa24736b5c709a5a60beb00, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/B/a2f611f82f4b48f0b1f66b1d3e98763f, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/B/21b806fc0f7f41c2b61745a3181c339e, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/B/c82b06a6aae44ad8a242e5105d206819, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/B/640431c713c94b3bb44acca27ba0b0ca, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/B/c7e598dcb45e40aea0c5899f5dd1c491, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/B/0a647c4a83754901a9fd449ebbc8364f, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/B/4a47a809cdf045b0bff0c9fd7cc0ce3a, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/B/d921f30e70c84bed960a37d28b43b8c4, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/B/2769b340de12499c9508aa15412f083a, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/B/40c2c459df1a4c26b6277e187e31a9cc, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/B/c8d0acf730a7459492179f5740a9235f, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/B/ae2b7c794b4a44f09914e452821e7dd2] to archive 2023-04-16 00:54:43,026 DEBUG [StoreCloser-TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-16 00:54:43,028 DEBUG [StoreCloser-TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/B/3f116ca5aea14ee089ffd2938d1a6e36 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/B/3f116ca5aea14ee089ffd2938d1a6e36 2023-04-16 00:54:43,029 DEBUG [StoreCloser-TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/B/004575861f934c2da4ab487a312b7e5d to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/B/004575861f934c2da4ab487a312b7e5d 2023-04-16 00:54:43,029 DEBUG [StoreCloser-TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/B/9d8eae1b34a04e2f9b603066470cb6ca to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/B/9d8eae1b34a04e2f9b603066470cb6ca 2023-04-16 00:54:43,030 DEBUG [StoreCloser-TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/B/a54d08fb83414d0f9473426c7209b4a4 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/B/a54d08fb83414d0f9473426c7209b4a4 2023-04-16 00:54:43,031 DEBUG [StoreCloser-TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/B/60b85be4841644a385e443b5d1e20625 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/B/60b85be4841644a385e443b5d1e20625 2023-04-16 00:54:43,032 DEBUG [StoreCloser-TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/B/4fe52a20c50b4913b5f13dd3ac7e3bb4 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/B/4fe52a20c50b4913b5f13dd3ac7e3bb4 2023-04-16 00:54:43,033 DEBUG [StoreCloser-TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/B/8e66594f3e484d3eab1c157a1b7e145d to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/B/8e66594f3e484d3eab1c157a1b7e145d 2023-04-16 00:54:43,034 DEBUG [StoreCloser-TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/B/84950f4f4c394a79867b48b57fbfbb7a to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/B/84950f4f4c394a79867b48b57fbfbb7a 2023-04-16 00:54:43,034 DEBUG [StoreCloser-TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/B/f02f9a14b5bb4aacaad5dc979ad36280 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/B/f02f9a14b5bb4aacaad5dc979ad36280 2023-04-16 00:54:43,035 DEBUG [StoreCloser-TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/B/05ae3848c7b54920b331066dbd534638 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/B/05ae3848c7b54920b331066dbd534638 2023-04-16 00:54:43,036 DEBUG [StoreCloser-TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/B/92aca36b949c48efbb8ad2d0834d0869 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/B/92aca36b949c48efbb8ad2d0834d0869 2023-04-16 00:54:43,037 DEBUG [StoreCloser-TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/B/0a265fceaa9f42b5a2112160556b7e93 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/B/0a265fceaa9f42b5a2112160556b7e93 2023-04-16 00:54:43,038 DEBUG [StoreCloser-TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/B/f8ed64d1e7e040a78b5f518f4babbe79 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/B/f8ed64d1e7e040a78b5f518f4babbe79 2023-04-16 00:54:43,039 DEBUG [StoreCloser-TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/B/cc2a6584442c4dada7e4049ae45d3ed9 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/B/cc2a6584442c4dada7e4049ae45d3ed9 2023-04-16 00:54:43,039 DEBUG [StoreCloser-TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/B/d6ae0991afa24736b5c709a5a60beb00 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/B/d6ae0991afa24736b5c709a5a60beb00 2023-04-16 00:54:43,040 DEBUG [StoreCloser-TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/B/a2f611f82f4b48f0b1f66b1d3e98763f to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/B/a2f611f82f4b48f0b1f66b1d3e98763f 2023-04-16 00:54:43,041 DEBUG [StoreCloser-TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/B/21b806fc0f7f41c2b61745a3181c339e to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/B/21b806fc0f7f41c2b61745a3181c339e 2023-04-16 00:54:43,042 DEBUG [StoreCloser-TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/B/c82b06a6aae44ad8a242e5105d206819 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/B/c82b06a6aae44ad8a242e5105d206819 2023-04-16 00:54:43,043 DEBUG [StoreCloser-TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/B/640431c713c94b3bb44acca27ba0b0ca to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/B/640431c713c94b3bb44acca27ba0b0ca 2023-04-16 00:54:43,043 DEBUG [StoreCloser-TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/B/c7e598dcb45e40aea0c5899f5dd1c491 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/B/c7e598dcb45e40aea0c5899f5dd1c491 2023-04-16 00:54:43,044 DEBUG [StoreCloser-TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/B/0a647c4a83754901a9fd449ebbc8364f to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/B/0a647c4a83754901a9fd449ebbc8364f 2023-04-16 00:54:43,045 DEBUG [StoreCloser-TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/B/4a47a809cdf045b0bff0c9fd7cc0ce3a to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/B/4a47a809cdf045b0bff0c9fd7cc0ce3a 2023-04-16 00:54:43,046 DEBUG [StoreCloser-TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/B/d921f30e70c84bed960a37d28b43b8c4 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/B/d921f30e70c84bed960a37d28b43b8c4 2023-04-16 00:54:43,047 DEBUG [StoreCloser-TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/B/2769b340de12499c9508aa15412f083a to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/B/2769b340de12499c9508aa15412f083a 2023-04-16 00:54:43,048 DEBUG [StoreCloser-TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/B/40c2c459df1a4c26b6277e187e31a9cc to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/B/40c2c459df1a4c26b6277e187e31a9cc 2023-04-16 00:54:43,049 DEBUG [StoreCloser-TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/B/c8d0acf730a7459492179f5740a9235f to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/B/c8d0acf730a7459492179f5740a9235f 2023-04-16 00:54:43,050 DEBUG [StoreCloser-TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/B/ae2b7c794b4a44f09914e452821e7dd2 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/B/ae2b7c794b4a44f09914e452821e7dd2 2023-04-16 00:54:43,052 DEBUG [StoreCloser-TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/C/61b84698f9a14cf0869de0b18914632a, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/C/f44186f6c28d4fce89bbc0768d661e1e, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/C/81819213e8eb40c39c93e8431207dc9d, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/C/3d7ec6f512f24453bc3ce42107d348ce, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/C/8eccc915c86748938f90f1b89ad3d918, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/C/ee0b7de082fa424ca99f0a881ada0ba5, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/C/b57b50aacd47434280ba1d9b02ca28fb, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/C/1426976e5f1b487c9b2a3b33ba577825, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/C/d499851602f249cf8e595a65615ad78e, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/C/f537bd4395704989b5500cab8788532e, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/C/f5360c0edde043ab9f9d1b3a2ee9b3ad, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/C/c3b7210eb7a940cea0dc85c6ec097c64, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/C/337e0546b2704a409812eaba605aee69, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/C/b8f3ee980a1c46929eae4e977b943d83, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/C/8a308f513dd94800b86ab205377d09a9, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/C/0576db228c8842cc83f5d67c893bc1c4, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/C/54170cd1ceda4274a259fe41ee4da8f8, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/C/c728232926ba42d0bc05b59493372d69, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/C/621791b9c43346cdbf24e4d25984a701, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/C/b6ec6a87ff8b445b94632270fc3614e5, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/C/463ab8e515bb48a9b12d7af4426426be, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/C/2c82f146691c43a09502c2c522f8b7cf, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/C/68c6722fdd444523ac341216c11dcdc4, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/C/224fa70428f84145a9412d779925f5b2, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/C/0f00134bae7c4109a12dd915c06cdbc0, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/C/55898da5367340c88a160eadeba65d99, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/C/f60febfbc08f4af5b4af6be1c8720f06] to archive 2023-04-16 00:54:43,053 DEBUG [StoreCloser-TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-16 00:54:43,054 DEBUG [StoreCloser-TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/C/61b84698f9a14cf0869de0b18914632a to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/C/61b84698f9a14cf0869de0b18914632a 2023-04-16 00:54:43,055 DEBUG [StoreCloser-TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/C/f44186f6c28d4fce89bbc0768d661e1e to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/C/f44186f6c28d4fce89bbc0768d661e1e 2023-04-16 00:54:43,056 DEBUG [StoreCloser-TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/C/81819213e8eb40c39c93e8431207dc9d to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/C/81819213e8eb40c39c93e8431207dc9d 2023-04-16 00:54:43,057 DEBUG [StoreCloser-TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/C/3d7ec6f512f24453bc3ce42107d348ce to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/C/3d7ec6f512f24453bc3ce42107d348ce 2023-04-16 00:54:43,058 DEBUG [StoreCloser-TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/C/8eccc915c86748938f90f1b89ad3d918 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/C/8eccc915c86748938f90f1b89ad3d918 2023-04-16 00:54:43,059 DEBUG [StoreCloser-TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/C/ee0b7de082fa424ca99f0a881ada0ba5 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/C/ee0b7de082fa424ca99f0a881ada0ba5 2023-04-16 00:54:43,059 DEBUG [StoreCloser-TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/C/b57b50aacd47434280ba1d9b02ca28fb to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/C/b57b50aacd47434280ba1d9b02ca28fb 2023-04-16 00:54:43,060 DEBUG [StoreCloser-TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/C/1426976e5f1b487c9b2a3b33ba577825 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/C/1426976e5f1b487c9b2a3b33ba577825 2023-04-16 00:54:43,061 DEBUG [StoreCloser-TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/C/d499851602f249cf8e595a65615ad78e to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/C/d499851602f249cf8e595a65615ad78e 2023-04-16 00:54:43,062 DEBUG [StoreCloser-TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/C/f537bd4395704989b5500cab8788532e to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/C/f537bd4395704989b5500cab8788532e 2023-04-16 00:54:43,063 DEBUG [StoreCloser-TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/C/f5360c0edde043ab9f9d1b3a2ee9b3ad to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/C/f5360c0edde043ab9f9d1b3a2ee9b3ad 2023-04-16 00:54:43,064 DEBUG [StoreCloser-TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/C/c3b7210eb7a940cea0dc85c6ec097c64 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/C/c3b7210eb7a940cea0dc85c6ec097c64 2023-04-16 00:54:43,065 DEBUG [StoreCloser-TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/C/337e0546b2704a409812eaba605aee69 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/C/337e0546b2704a409812eaba605aee69 2023-04-16 00:54:43,066 DEBUG [StoreCloser-TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/C/b8f3ee980a1c46929eae4e977b943d83 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/C/b8f3ee980a1c46929eae4e977b943d83 2023-04-16 00:54:43,067 DEBUG [StoreCloser-TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/C/8a308f513dd94800b86ab205377d09a9 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/C/8a308f513dd94800b86ab205377d09a9 2023-04-16 00:54:43,069 DEBUG [StoreCloser-TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/C/0576db228c8842cc83f5d67c893bc1c4 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/C/0576db228c8842cc83f5d67c893bc1c4 2023-04-16 00:54:43,070 DEBUG [StoreCloser-TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/C/54170cd1ceda4274a259fe41ee4da8f8 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/C/54170cd1ceda4274a259fe41ee4da8f8 2023-04-16 00:54:43,071 DEBUG [StoreCloser-TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/C/c728232926ba42d0bc05b59493372d69 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/C/c728232926ba42d0bc05b59493372d69 2023-04-16 00:54:43,072 DEBUG [StoreCloser-TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/C/621791b9c43346cdbf24e4d25984a701 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/C/621791b9c43346cdbf24e4d25984a701 2023-04-16 00:54:43,073 DEBUG [StoreCloser-TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/C/b6ec6a87ff8b445b94632270fc3614e5 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/C/b6ec6a87ff8b445b94632270fc3614e5 2023-04-16 00:54:43,074 DEBUG [StoreCloser-TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/C/463ab8e515bb48a9b12d7af4426426be to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/C/463ab8e515bb48a9b12d7af4426426be 2023-04-16 00:54:43,075 DEBUG [StoreCloser-TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/C/2c82f146691c43a09502c2c522f8b7cf to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/C/2c82f146691c43a09502c2c522f8b7cf 2023-04-16 00:54:43,077 DEBUG [StoreCloser-TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/C/68c6722fdd444523ac341216c11dcdc4 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/C/68c6722fdd444523ac341216c11dcdc4 2023-04-16 00:54:43,078 DEBUG [StoreCloser-TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/C/224fa70428f84145a9412d779925f5b2 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/C/224fa70428f84145a9412d779925f5b2 2023-04-16 00:54:43,079 DEBUG [StoreCloser-TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/C/0f00134bae7c4109a12dd915c06cdbc0 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/C/0f00134bae7c4109a12dd915c06cdbc0 2023-04-16 00:54:43,080 DEBUG [StoreCloser-TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/C/55898da5367340c88a160eadeba65d99 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/C/55898da5367340c88a160eadeba65d99 2023-04-16 00:54:43,081 DEBUG [StoreCloser-TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/C/f60febfbc08f4af5b4af6be1c8720f06 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/C/f60febfbc08f4af5b4af6be1c8720f06 2023-04-16 00:54:43,086 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/recovered.edits/524.seqid, newMaxSeqId=524, maxSeqId=4 2023-04-16 00:54:43,088 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f. 2023-04-16 00:54:43,088 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for bcfff279d7c9f90f5c57aef887110e5f: 2023-04-16 00:54:43,089 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed bcfff279d7c9f90f5c57aef887110e5f 2023-04-16 00:54:43,090 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=25 updating hbase:meta row=bcfff279d7c9f90f5c57aef887110e5f, regionState=CLOSED 2023-04-16 00:54:43,090 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681606483090"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681606483090"}]},"ts":"1681606483090"} 2023-04-16 00:54:43,092 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=26, resume processing ppid=25 2023-04-16 00:54:43,092 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=26, ppid=25, state=SUCCESS; CloseRegionProcedure bcfff279d7c9f90f5c57aef887110e5f, server=jenkins-hbase4.apache.org,34975,1681606419217 in 1.9140 sec 2023-04-16 00:54:43,093 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=25, resume processing ppid=24 2023-04-16 00:54:43,093 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=25, ppid=24, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=bcfff279d7c9f90f5c57aef887110e5f, UNASSIGN in 1.9190 sec 2023-04-16 00:54:43,093 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681606483093"}]},"ts":"1681606483093"} 2023-04-16 00:54:43,094 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-16 00:54:43,096 INFO [PEWorker-5] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-16 00:54:43,097 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=24, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 1.9310 sec 2023-04-16 00:54:43,273 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-16 00:54:43,273 INFO [Listener at localhost/33477] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 24 completed 2023-04-16 00:54:43,274 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete TestAcidGuarantees 2023-04-16 00:54:43,275 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure2.ProcedureExecutor(1029): Stored pid=27, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-16 00:54:43,276 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=27, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-16 00:54:43,277 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1227): Checking to see if procedure is done pid=27 2023-04-16 00:54:43,277 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=27, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-16 00:54:43,281 DEBUG [HFileArchiver-5] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/.tmp/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f 2023-04-16 00:54:43,284 DEBUG [HFileArchiver-5] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/.tmp/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/A, FileablePath, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/.tmp/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/B, FileablePath, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/.tmp/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/C, FileablePath, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/.tmp/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/recovered.edits] 2023-04-16 00:54:43,286 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/.tmp/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/A/051d11b1c1004549ba4b17546e17c613 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/A/051d11b1c1004549ba4b17546e17c613 2023-04-16 00:54:43,287 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/.tmp/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/A/0b2360c43fd64951ab6f66ac174d825f to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/A/0b2360c43fd64951ab6f66ac174d825f 2023-04-16 00:54:43,288 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/.tmp/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/A/8d86a17a3e8c49cdb362ff8c36df9f9b to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/A/8d86a17a3e8c49cdb362ff8c36df9f9b 2023-04-16 00:54:43,290 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/.tmp/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/B/11ed392fa0f9417885e48be6071f9676 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/B/11ed392fa0f9417885e48be6071f9676 2023-04-16 00:54:43,291 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/.tmp/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/B/1bfa6c62814b4968ba4d40f18210fc55 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/B/1bfa6c62814b4968ba4d40f18210fc55 2023-04-16 00:54:43,292 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/.tmp/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/B/cd7f6fe0d360478ab00ecd5d819094f2 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/B/cd7f6fe0d360478ab00ecd5d819094f2 2023-04-16 00:54:43,294 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/.tmp/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/C/03355f8c3d4f4994bc4850a253c0f75e to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/C/03355f8c3d4f4994bc4850a253c0f75e 2023-04-16 00:54:43,295 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/.tmp/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/C/0c61ec02d47746798df819782d4126a4 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/C/0c61ec02d47746798df819782d4126a4 2023-04-16 00:54:43,297 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/.tmp/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/C/a86fe8878e3d4cd6aa412f7eae479c31 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/C/a86fe8878e3d4cd6aa412f7eae479c31 2023-04-16 00:54:43,299 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/.tmp/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/recovered.edits/524.seqid to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f/recovered.edits/524.seqid 2023-04-16 00:54:43,300 DEBUG [HFileArchiver-5] backup.HFileArchiver(596): Deleted hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/.tmp/data/default/TestAcidGuarantees/bcfff279d7c9f90f5c57aef887110e5f 2023-04-16 00:54:43,300 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-16 00:54:43,301 DEBUG [PEWorker-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-16 00:54:43,301 DEBUG [PEWorker-1] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A] 2023-04-16 00:54:43,304 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041612f7b8e0a2be4939ad59eb743a1b227e to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041612f7b8e0a2be4939ad59eb743a1b227e 2023-04-16 00:54:43,305 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230416224b4fc0eba64046ab600d8bf992877f to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230416224b4fc0eba64046ab600d8bf992877f 2023-04-16 00:54:43,306 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304162f4daf5a859d4f1d8c9b6babea9cfc01 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304162f4daf5a859d4f1d8c9b6babea9cfc01 2023-04-16 00:54:43,307 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041631de60ca1aea4853bbb7eb2179df4428 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041631de60ca1aea4853bbb7eb2179df4428 2023-04-16 00:54:43,308 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041633808f02c44b489d8d341c40fb5b51e3 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041633808f02c44b489d8d341c40fb5b51e3 2023-04-16 00:54:43,309 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304164b0ae11fe269429db93855bf9ecb3a69 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304164b0ae11fe269429db93855bf9ecb3a69 2023-04-16 00:54:43,310 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230416528696a9e5b748f98bb9cbf10b022cd7 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230416528696a9e5b748f98bb9cbf10b022cd7 2023-04-16 00:54:43,311 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230416610afa9caefe4f93866d48308c418c91 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230416610afa9caefe4f93866d48308c418c91 2023-04-16 00:54:43,312 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230416628c463dccfc4a7c980bd82f6622039f to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230416628c463dccfc4a7c980bd82f6622039f 2023-04-16 00:54:43,313 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304166a7f1e1d967940e097800985bd42141a to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304166a7f1e1d967940e097800985bd42141a 2023-04-16 00:54:43,314 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304167613a856bc244e709162d9d798212feb to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304167613a856bc244e709162d9d798212feb 2023-04-16 00:54:43,315 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041676f3e7bb794e4a368c5742d28ae7215c to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041676f3e7bb794e4a368c5742d28ae7215c 2023-04-16 00:54:43,316 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304168becbef46c2d43dd896d4c3646710d70 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304168becbef46c2d43dd896d4c3646710d70 2023-04-16 00:54:43,318 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230416a0eaafc55a8443a381bb0fff82d118ea to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230416a0eaafc55a8443a381bb0fff82d118ea 2023-04-16 00:54:43,319 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230416a1d3939c9b71438a952377e0262c549e to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230416a1d3939c9b71438a952377e0262c549e 2023-04-16 00:54:43,320 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230416a81b747b432544c181065018b419dd26 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230416a81b747b432544c181065018b419dd26 2023-04-16 00:54:43,321 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230416b5a9deceffe44e4280134c07b9d47620 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230416b5a9deceffe44e4280134c07b9d47620 2023-04-16 00:54:43,322 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230416bdcdc451a7e5498594cd9c2a8d5497c8 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230416bdcdc451a7e5498594cd9c2a8d5497c8 2023-04-16 00:54:43,323 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230416ce31c3f3754346928bda7ba990b835e0 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230416ce31c3f3754346928bda7ba990b835e0 2023-04-16 00:54:43,324 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230416d67a5ffb9cb148e1ab9ada4c26e28412 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230416d67a5ffb9cb148e1ab9ada4c26e28412 2023-04-16 00:54:43,325 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230416e24073e8b2b044508bc4a46d85ba5881 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230416e24073e8b2b044508bc4a46d85ba5881 2023-04-16 00:54:43,326 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230416e2582097f7b44031b681effb499a685c to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230416e2582097f7b44031b681effb499a685c 2023-04-16 00:54:43,327 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230416ef9160b9b3b740cbb675ced437274422 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230416ef9160b9b3b740cbb675ced437274422 2023-04-16 00:54:43,329 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230416f102ed9292df49528bff1ce49ec69238 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230416f102ed9292df49528bff1ce49ec69238 2023-04-16 00:54:43,329 DEBUG [PEWorker-1] backup.HFileArchiver(596): Deleted hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-16 00:54:43,331 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=27, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-16 00:54:43,333 WARN [PEWorker-1] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-16 00:54:43,335 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-16 00:54:43,336 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=27, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-16 00:54:43,336 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-16 00:54:43,336 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1681606483336"}]},"ts":"9223372036854775807"} 2023-04-16 00:54:43,337 INFO [PEWorker-1] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-16 00:54:43,337 DEBUG [PEWorker-1] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => bcfff279d7c9f90f5c57aef887110e5f, NAME => 'TestAcidGuarantees,,1681606454428.bcfff279d7c9f90f5c57aef887110e5f.', STARTKEY => '', ENDKEY => ''}] 2023-04-16 00:54:43,337 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-16 00:54:43,338 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1681606483337"}]},"ts":"9223372036854775807"} 2023-04-16 00:54:43,339 INFO [PEWorker-1] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-16 00:54:43,340 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(130): Finished pid=27, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-16 00:54:43,341 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=27, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 67 msec 2023-04-16 00:54:43,377 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1227): Checking to see if procedure is done pid=27 2023-04-16 00:54:43,378 INFO [Listener at localhost/33477] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 27 completed 2023-04-16 00:54:43,389 INFO [Listener at localhost/33477] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testMobMixedAtomicity Thread=229 (was 230), OpenFileDescriptor=545 (was 575), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=337 (was 334) - SystemLoadAverage LEAK? -, ProcessCount=171 (was 171), AvailableMemoryMB=786 (was 878) 2023-04-16 00:54:43,399 INFO [Listener at localhost/33477] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testGetAtomicity Thread=229, OpenFileDescriptor=545, MaxFileDescriptor=60000, SystemLoadAverage=337, ProcessCount=171, AvailableMemoryMB=786 2023-04-16 00:54:43,401 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-16 00:54:43,401 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-16 00:54:43,402 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure2.ProcedureExecutor(1029): Stored pid=28, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-16 00:54:43,404 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-16 00:54:43,404 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 28 2023-04-16 00:54:43,404 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-16 00:54:43,404 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-04-16 00:54:43,406 DEBUG [HFileArchiver-6] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/.tmp/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08 2023-04-16 00:54:43,406 DEBUG [HFileArchiver-6] backup.HFileArchiver(153): Directory hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/.tmp/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08 empty. 2023-04-16 00:54:43,406 DEBUG [HFileArchiver-6] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/.tmp/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08 2023-04-16 00:54:43,406 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-16 00:54:43,505 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-04-16 00:54:43,706 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-04-16 00:54:43,814 DEBUG [PEWorker-3] util.FSTableDescriptors(570): Wrote into hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-16 00:54:43,815 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => 5dcf6997c5a081edf1d95a6a373c6b08, NAME => 'TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/.tmp 2023-04-16 00:54:44,007 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-04-16 00:54:44,221 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-16 00:54:44,221 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing 5dcf6997c5a081edf1d95a6a373c6b08, disabling compactions & flushes 2023-04-16 00:54:44,221 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. 2023-04-16 00:54:44,221 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. 2023-04-16 00:54:44,222 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. after waiting 0 ms 2023-04-16 00:54:44,222 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. 2023-04-16 00:54:44,222 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. 2023-04-16 00:54:44,222 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for 5dcf6997c5a081edf1d95a6a373c6b08: 2023-04-16 00:54:44,224 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-16 00:54:44,225 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681606484225"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681606484225"}]},"ts":"1681606484225"} 2023-04-16 00:54:44,226 INFO [PEWorker-3] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-16 00:54:44,226 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-16 00:54:44,227 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681606484226"}]},"ts":"1681606484226"} 2023-04-16 00:54:44,227 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-16 00:54:44,231 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=29, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=5dcf6997c5a081edf1d95a6a373c6b08, ASSIGN}] 2023-04-16 00:54:44,233 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=29, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=5dcf6997c5a081edf1d95a6a373c6b08, ASSIGN 2023-04-16 00:54:44,233 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=29, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=5dcf6997c5a081edf1d95a6a373c6b08, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,34975,1681606419217; forceNewPlan=false, retain=false 2023-04-16 00:54:44,384 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=29 updating hbase:meta row=5dcf6997c5a081edf1d95a6a373c6b08, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:44,384 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681606484384"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681606484384"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681606484384"}]},"ts":"1681606484384"} 2023-04-16 00:54:44,386 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=30, ppid=29, state=RUNNABLE; OpenRegionProcedure 5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217}] 2023-04-16 00:54:44,507 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-04-16 00:54:44,540 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. 2023-04-16 00:54:44,540 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 5dcf6997c5a081edf1d95a6a373c6b08, NAME => 'TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08.', STARTKEY => '', ENDKEY => ''} 2023-04-16 00:54:44,540 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 5dcf6997c5a081edf1d95a6a373c6b08 2023-04-16 00:54:44,541 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-16 00:54:44,541 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 5dcf6997c5a081edf1d95a6a373c6b08 2023-04-16 00:54:44,541 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 5dcf6997c5a081edf1d95a6a373c6b08 2023-04-16 00:54:44,542 INFO [StoreOpener-5dcf6997c5a081edf1d95a6a373c6b08-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 5dcf6997c5a081edf1d95a6a373c6b08 2023-04-16 00:54:44,543 DEBUG [StoreOpener-5dcf6997c5a081edf1d95a6a373c6b08-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/A 2023-04-16 00:54:44,544 DEBUG [StoreOpener-5dcf6997c5a081edf1d95a6a373c6b08-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/A 2023-04-16 00:54:44,544 INFO [StoreOpener-5dcf6997c5a081edf1d95a6a373c6b08-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-16 00:54:44,544 INFO [StoreOpener-5dcf6997c5a081edf1d95a6a373c6b08-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 5dcf6997c5a081edf1d95a6a373c6b08 columnFamilyName A 2023-04-16 00:54:44,545 INFO [StoreOpener-5dcf6997c5a081edf1d95a6a373c6b08-1] regionserver.HStore(310): Store=5dcf6997c5a081edf1d95a6a373c6b08/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-16 00:54:44,545 INFO [StoreOpener-5dcf6997c5a081edf1d95a6a373c6b08-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 5dcf6997c5a081edf1d95a6a373c6b08 2023-04-16 00:54:44,546 DEBUG [StoreOpener-5dcf6997c5a081edf1d95a6a373c6b08-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/B 2023-04-16 00:54:44,546 DEBUG [StoreOpener-5dcf6997c5a081edf1d95a6a373c6b08-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/B 2023-04-16 00:54:44,546 INFO [StoreOpener-5dcf6997c5a081edf1d95a6a373c6b08-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-16 00:54:44,546 INFO [StoreOpener-5dcf6997c5a081edf1d95a6a373c6b08-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 5dcf6997c5a081edf1d95a6a373c6b08 columnFamilyName B 2023-04-16 00:54:44,546 INFO [StoreOpener-5dcf6997c5a081edf1d95a6a373c6b08-1] regionserver.HStore(310): Store=5dcf6997c5a081edf1d95a6a373c6b08/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-16 00:54:44,547 INFO [StoreOpener-5dcf6997c5a081edf1d95a6a373c6b08-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 5dcf6997c5a081edf1d95a6a373c6b08 2023-04-16 00:54:44,547 DEBUG [StoreOpener-5dcf6997c5a081edf1d95a6a373c6b08-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/C 2023-04-16 00:54:44,548 DEBUG [StoreOpener-5dcf6997c5a081edf1d95a6a373c6b08-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/C 2023-04-16 00:54:44,548 INFO [StoreOpener-5dcf6997c5a081edf1d95a6a373c6b08-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-16 00:54:44,548 INFO [StoreOpener-5dcf6997c5a081edf1d95a6a373c6b08-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 5dcf6997c5a081edf1d95a6a373c6b08 columnFamilyName C 2023-04-16 00:54:44,548 INFO [StoreOpener-5dcf6997c5a081edf1d95a6a373c6b08-1] regionserver.HStore(310): Store=5dcf6997c5a081edf1d95a6a373c6b08/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-16 00:54:44,548 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. 2023-04-16 00:54:44,549 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08 2023-04-16 00:54:44,549 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08 2023-04-16 00:54:44,551 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-16 00:54:44,552 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 5dcf6997c5a081edf1d95a6a373c6b08 2023-04-16 00:54:44,553 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-16 00:54:44,554 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 5dcf6997c5a081edf1d95a6a373c6b08; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=60755558, jitterRate=-0.09467163681983948}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-16 00:54:44,554 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 5dcf6997c5a081edf1d95a6a373c6b08: 2023-04-16 00:54:44,554 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08., pid=30, masterSystemTime=1681606484537 2023-04-16 00:54:44,556 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. 2023-04-16 00:54:44,556 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. 2023-04-16 00:54:44,556 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=29 updating hbase:meta row=5dcf6997c5a081edf1d95a6a373c6b08, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:44,556 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681606484556"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1681606484556"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1681606484556"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1681606484556"}]},"ts":"1681606484556"} 2023-04-16 00:54:44,558 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=30, resume processing ppid=29 2023-04-16 00:54:44,558 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=30, ppid=29, state=SUCCESS; OpenRegionProcedure 5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 in 171 msec 2023-04-16 00:54:44,559 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=29, resume processing ppid=28 2023-04-16 00:54:44,559 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=29, ppid=28, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=5dcf6997c5a081edf1d95a6a373c6b08, ASSIGN in 327 msec 2023-04-16 00:54:44,560 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-16 00:54:44,560 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681606484560"}]},"ts":"1681606484560"} 2023-04-16 00:54:44,561 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-16 00:54:44,563 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-16 00:54:44,564 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=28, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 1.1610 sec 2023-04-16 00:54:45,508 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-04-16 00:54:45,508 INFO [Listener at localhost/33477] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 28 completed 2023-04-16 00:54:45,509 DEBUG [Listener at localhost/33477] zookeeper.ReadOnlyZKClient(139): Connect 0x12e82edd to 127.0.0.1:53150 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-16 00:54:45,514 DEBUG [Listener at localhost/33477] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7ed1dde9, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-16 00:54:45,515 DEBUG [Listener at localhost/33477] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-16 00:54:45,516 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:58248, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-16 00:54:45,517 DEBUG [Listener at localhost/33477] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-16 00:54:45,519 INFO [RS-EventLoopGroup-1-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:35056, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-16 00:54:45,520 DEBUG [Listener at localhost/33477] zookeeper.ReadOnlyZKClient(139): Connect 0x23c97092 to 127.0.0.1:53150 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-16 00:54:45,525 DEBUG [Listener at localhost/33477] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@70043cf0, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-16 00:54:45,525 DEBUG [Listener at localhost/33477] zookeeper.ReadOnlyZKClient(139): Connect 0x5c124820 to 127.0.0.1:53150 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-16 00:54:45,534 DEBUG [Listener at localhost/33477] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@34b7fed0, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-16 00:54:45,534 DEBUG [Listener at localhost/33477] zookeeper.ReadOnlyZKClient(139): Connect 0x3157d4df to 127.0.0.1:53150 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-16 00:54:45,538 DEBUG [Listener at localhost/33477] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7f4ad99d, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-16 00:54:45,538 DEBUG [Listener at localhost/33477] zookeeper.ReadOnlyZKClient(139): Connect 0x3cf5bdb7 to 127.0.0.1:53150 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-16 00:54:45,541 DEBUG [Listener at localhost/33477] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@330b77da, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-16 00:54:45,541 DEBUG [Listener at localhost/33477] zookeeper.ReadOnlyZKClient(139): Connect 0x6919b2a0 to 127.0.0.1:53150 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-16 00:54:45,545 DEBUG [Listener at localhost/33477] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7810a231, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-16 00:54:45,546 DEBUG [Listener at localhost/33477] zookeeper.ReadOnlyZKClient(139): Connect 0x08cd7cb0 to 127.0.0.1:53150 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-16 00:54:45,549 DEBUG [Listener at localhost/33477] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@753156b2, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-16 00:54:45,549 DEBUG [Listener at localhost/33477] zookeeper.ReadOnlyZKClient(139): Connect 0x0ea1ca03 to 127.0.0.1:53150 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-16 00:54:45,552 DEBUG [Listener at localhost/33477] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6dea0f9b, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-16 00:54:45,552 DEBUG [Listener at localhost/33477] zookeeper.ReadOnlyZKClient(139): Connect 0x433f251d to 127.0.0.1:53150 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-16 00:54:45,556 DEBUG [Listener at localhost/33477] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@434e0eab, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-16 00:54:45,556 DEBUG [Listener at localhost/33477] zookeeper.ReadOnlyZKClient(139): Connect 0x66ee0bd0 to 127.0.0.1:53150 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-16 00:54:45,561 DEBUG [Listener at localhost/33477] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5c43dddd, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-16 00:54:45,562 DEBUG [Listener at localhost/33477] zookeeper.ReadOnlyZKClient(139): Connect 0x3b4658f8 to 127.0.0.1:53150 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-16 00:54:45,565 DEBUG [Listener at localhost/33477] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2de064cb, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-16 00:54:45,573 DEBUG [hconnection-0x6aca1a18-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-16 00:54:45,574 DEBUG [hconnection-0x2239a362-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-16 00:54:45,575 DEBUG [hconnection-0x24396a18-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-16 00:54:45,575 DEBUG [hconnection-0x5ba1e001-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-16 00:54:45,575 DEBUG [hconnection-0x4d8276c5-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-16 00:54:45,575 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:54:45,577 DEBUG [hconnection-0x1e64b252-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-16 00:54:45,577 DEBUG [hconnection-0x2b7ff742-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-16 00:54:45,577 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:58266, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-16 00:54:45,578 DEBUG [hconnection-0x12304985-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-16 00:54:45,578 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:58292, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-16 00:54:45,578 DEBUG [hconnection-0x168a64f8-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-16 00:54:45,579 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:58336, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-16 00:54:45,581 DEBUG [hconnection-0x7733738a-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-16 00:54:45,583 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:58264, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-16 00:54:45,583 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:58252, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-16 00:54:45,583 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:58284, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-16 00:54:45,583 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:58282, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-16 00:54:45,583 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:58322, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-16 00:54:45,584 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:58306, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-16 00:54:45,587 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:54:45,588 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:58346, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-16 00:54:45,588 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:54:45,588 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:54:45,588 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:54:45,589 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:54:45,589 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:54:45,592 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:45,592 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:45,597 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:45,597 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:54:45,597 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:54:45,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:54:45,597 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:45,597 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:54:45,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:45,598 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:45,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:54:45,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:45,598 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:54:45,604 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:54:45,604 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(9158): Flush requested on 5dcf6997c5a081edf1d95a6a373c6b08 2023-04-16 00:54:45,604 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 5dcf6997c5a081edf1d95a6a373c6b08 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-16 00:54:45,604 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:54:45,605 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:54:45,605 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:54:45,605 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:54:45,605 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:54:45,605 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:54:45,605 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. 2023-04-16 00:54:45,605 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. started... 2023-04-16 00:54:45,605 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. as already flushing 2023-04-16 00:54:45,605 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. 2023-04-16 00:54:45,605 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:54:45,605 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:54:45,605 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5dcf6997c5a081edf1d95a6a373c6b08, store=A 2023-04-16 00:54:45,605 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:45,606 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:54:45,606 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:54:45,606 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:54:45,606 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5dcf6997c5a081edf1d95a6a373c6b08, store=B 2023-04-16 00:54:45,606 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:54:45,606 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5dcf6997c5a081edf1d95a6a373c6b08, store=C 2023-04-16 00:54:45,606 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:54:45,607 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:45,608 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:45,608 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:54:45,608 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:45,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:45,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:45,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:45,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:45,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:45,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:45,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:45,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:45,611 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:54:45,611 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5f83693b[Count = 0] remaining members to acquire global barrier 2023-04-16 00:54:45,611 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:54:45,611 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:45,614 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:45,614 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:54:45,616 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:45,620 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:45,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:45,620 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:54:45,620 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:54:45,620 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:54:45,622 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:54:45,622 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:45,622 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:54:45,622 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:54:45,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:45,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:45,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:45,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:45,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:45,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:45,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:45,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:45,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:45,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:45,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:54:45,625 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:54:45,625 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:54:45,625 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:54:45,625 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:54:45,625 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:54:45,625 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:54:45,627 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:45,627 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:54:45,627 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:45,627 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:45,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:45,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:45,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:45,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:45,628 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:54:45,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:54:45,628 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:45,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:45,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:45,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:45,630 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:45,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:45,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:45,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:45,631 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:45,631 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58336 deadline: 1681606545626, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:45,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:45,631 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:45,631 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 6 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58282 deadline: 1681606545626, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:45,631 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:45,632 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58322 deadline: 1681606545626, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:45,635 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:45,635 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:45,635 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58306 deadline: 1681606545631, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:45,635 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58292 deadline: 1681606545632, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:45,636 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:45,636 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:45,636 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:45,636 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:45,636 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:45,636 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:45,636 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:45,636 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:54:45,636 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:54:45,636 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:54:45,637 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:54:45,636 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:54:45,637 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:54:45,636 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:54:45,637 DEBUG [Thread-1736] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:54:45,637 DEBUG [Thread-1736] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:54:45,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:54:45,637 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:54:45,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:54:45,637 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=14 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/.tmp/A/234b83427e2c4b31b199e2d41ca9a8b7 2023-04-16 00:54:45,690 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=14 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/.tmp/B/1a1b8c00fba64695895917834d3a77c5 2023-04-16 00:54:45,733 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:45,734 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58336 deadline: 1681606545732, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:45,734 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:45,734 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58282 deadline: 1681606545732, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:45,734 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:45,734 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58322 deadline: 1681606545732, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:45,737 DEBUG [Thread-1736] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:54:45,737 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:45,738 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58306 deadline: 1681606545736, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:45,738 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:45,738 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:54:45,738 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58292 deadline: 1681606545736, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:45,739 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:54:45,740 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:54:45,741 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:54:45,741 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:54:45,741 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:54:45,741 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:54:45,741 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:54:45,741 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:45,741 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:45,742 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:45,742 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:54:45,742 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:54:45,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:54:45,743 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:45,743 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:54:45,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:45,743 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:45,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:54:45,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:45,743 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:54:45,744 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:54:45,744 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:54:45,744 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:54:45,744 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:54:45,744 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:54:45,748 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:54:45,748 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:54:45,748 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. 2023-04-16 00:54:45,748 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. started... 2023-04-16 00:54:45,748 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. as already flushing 2023-04-16 00:54:45,748 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. 2023-04-16 00:54:45,748 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:54:45,748 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:54:45,748 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:45,748 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:54:45,748 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:54:45,750 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:45,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:45,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:45,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:45,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:45,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:45,750 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:45,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:45,751 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:45,751 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:54:45,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:45,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:45,751 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:54:45,752 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@24d2aca0[Count = 0] remaining members to acquire global barrier 2023-04-16 00:54:45,752 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:54:45,752 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:45,753 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:45,753 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:45,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:45,753 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:54:45,753 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:54:45,753 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:54:45,753 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:45,753 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:54:45,755 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:45,755 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:54:45,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:45,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:45,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:45,755 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:54:45,755 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:54:45,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:45,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:45,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:45,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:45,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:45,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:45,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:45,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:54:45,758 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:54:45,758 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:54:45,758 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:54:45,758 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:54:45,758 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:54:45,758 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:54:45,759 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:45,759 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:45,759 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:45,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:45,759 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:54:45,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:45,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:45,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:45,760 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:54:45,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:54:45,760 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:45,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:45,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:45,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:45,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:45,761 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:45,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:45,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:45,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:45,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:45,763 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:45,763 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:45,763 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:45,763 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:54:45,763 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:54:45,763 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:45,764 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:45,764 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:45,764 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:45,764 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:54:45,764 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:54:45,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:54:45,764 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:54:45,764 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:54:45,764 DEBUG [Thread-1736] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:54:45,764 DEBUG [Thread-1736] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:54:45,764 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:54:45,764 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:54:45,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:54:45,864 DEBUG [Thread-1736] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:54:45,865 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:54:45,866 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:54:45,867 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:54:45,867 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:54:45,867 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:54:45,867 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:54:45,867 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:54:45,867 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:54:45,868 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:45,868 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:45,869 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:54:45,869 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:54:45,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:54:45,869 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:45,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:45,870 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:45,870 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:54:45,870 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:45,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:54:45,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:45,870 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:54:45,870 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:54:45,870 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:54:45,870 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:54:45,871 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:54:45,871 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:54:45,871 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:54:45,871 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:54:45,871 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. 2023-04-16 00:54:45,871 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. started... 2023-04-16 00:54:45,871 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. as already flushing 2023-04-16 00:54:45,871 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. 2023-04-16 00:54:45,871 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:54:45,871 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:54:45,871 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:45,871 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:54:45,871 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:54:45,873 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:45,873 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:45,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:45,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:45,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:45,874 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:45,874 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:54:45,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:45,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:45,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:45,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:45,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:45,875 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:54:45,875 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@47de01ee[Count = 0] remaining members to acquire global barrier 2023-04-16 00:54:45,875 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:54:45,875 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:45,876 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:45,876 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:45,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:45,876 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:54:45,876 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:54:45,876 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:45,876 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:54:45,876 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:54:45,878 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:45,878 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:54:45,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:45,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:45,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:45,878 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:54:45,878 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:54:45,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:45,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:45,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:45,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:45,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:45,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:45,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:45,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:54:45,880 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:54:45,880 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:54:45,880 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:54:45,880 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:54:45,880 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:54:45,880 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:54:45,882 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:45,882 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:45,882 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:54:45,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:45,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:45,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:45,882 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:45,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:45,883 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:45,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:45,883 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:54:45,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:45,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:54:45,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:45,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:45,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:45,884 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:45,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:45,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:45,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:45,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:45,892 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:45,892 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:45,892 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:54:45,892 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:45,892 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:54:45,892 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:54:45,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:54:45,892 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:54:45,892 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:54:45,892 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:54:45,892 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:54:45,892 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:45,893 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:45,893 DEBUG [Thread-1736] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:54:45,893 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:45,893 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:54:45,893 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:45,893 DEBUG [Thread-1736] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:54:45,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:54:45,936 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:45,936 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58282 deadline: 1681606545935, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:45,937 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:45,937 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:45,938 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58336 deadline: 1681606545935, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:45,938 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58322 deadline: 1681606545936, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:45,940 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:45,940 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58292 deadline: 1681606545939, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:45,940 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:45,941 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58306 deadline: 1681606545939, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:45,993 DEBUG [Thread-1736] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:54:45,993 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:54:45,995 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:54:45,996 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:54:45,996 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:54:45,996 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:54:45,996 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:54:45,996 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:54:45,996 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:54:45,996 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:45,996 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:45,997 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:54:45,997 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:45,997 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:54:45,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:54:45,998 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:45,998 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:54:45,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:45,998 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:45,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:54:45,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:45,998 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:54:45,999 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:54:45,999 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:54:45,999 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:54:45,999 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:54:45,999 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:54:46,001 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:54:46,001 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:54:46,001 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. 2023-04-16 00:54:46,001 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. started... 2023-04-16 00:54:46,001 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. as already flushing 2023-04-16 00:54:46,001 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. 2023-04-16 00:54:46,001 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:54:46,001 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:54:46,001 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:46,001 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:54:46,001 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:54:46,003 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:46,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:46,003 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:46,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:46,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:46,003 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:46,003 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:54:46,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:46,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:46,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:46,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:46,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:46,004 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:54:46,004 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1def70c8[Count = 0] remaining members to acquire global barrier 2023-04-16 00:54:46,004 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:54:46,004 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:46,005 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:46,005 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:46,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:46,006 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:54:46,006 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:54:46,006 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:46,006 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:54:46,006 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:54:46,008 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:54:46,008 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:54:46,008 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:46,008 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:54:46,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:46,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:46,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:46,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:46,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:46,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:46,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:46,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:46,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:46,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:46,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:54:46,010 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:54:46,010 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:54:46,010 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:54:46,010 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:54:46,010 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:54:46,010 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:54:46,012 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:46,012 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:46,012 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:46,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:46,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:46,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:46,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:46,012 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:54:46,012 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:54:46,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:54:46,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:46,013 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:46,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:46,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:46,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:46,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:46,014 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:46,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:46,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:46,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:46,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:46,018 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:46,018 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:46,018 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:46,018 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:46,018 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:54:46,018 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:46,018 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:54:46,018 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:46,018 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:54:46,018 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:54:46,018 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:46,018 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:54:46,018 DEBUG [Thread-1736] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:54:46,019 DEBUG [Thread-1736] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:54:46,018 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:54:46,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:54:46,019 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:54:46,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:54:46,110 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=14 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/.tmp/C/90cf6ae6569f4bc6a55737127551399f 2023-04-16 00:54:46,115 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/.tmp/A/234b83427e2c4b31b199e2d41ca9a8b7 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/A/234b83427e2c4b31b199e2d41ca9a8b7 2023-04-16 00:54:46,119 DEBUG [Thread-1736] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:54:46,119 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:54:46,120 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/A/234b83427e2c4b31b199e2d41ca9a8b7, entries=200, sequenceid=14, filesize=13.9 K 2023-04-16 00:54:46,120 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/.tmp/B/1a1b8c00fba64695895917834d3a77c5 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/B/1a1b8c00fba64695895917834d3a77c5 2023-04-16 00:54:46,121 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:54:46,122 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:54:46,122 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:54:46,122 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:54:46,122 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:54:46,122 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:54:46,122 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:54:46,122 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:46,122 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:46,124 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/B/1a1b8c00fba64695895917834d3a77c5, entries=150, sequenceid=14, filesize=11.6 K 2023-04-16 00:54:46,124 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:54:46,124 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:54:46,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:54:46,125 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/.tmp/C/90cf6ae6569f4bc6a55737127551399f as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/C/90cf6ae6569f4bc6a55737127551399f 2023-04-16 00:54:46,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:46,125 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:46,125 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:46,125 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:46,125 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:54:46,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:54:46,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:46,125 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:54:46,126 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:54:46,126 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:54:46,126 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:54:46,126 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:54:46,126 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:54:46,128 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/C/90cf6ae6569f4bc6a55737127551399f, entries=150, sequenceid=14, filesize=11.6 K 2023-04-16 00:54:46,128 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:54:46,128 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:54:46,128 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. 2023-04-16 00:54:46,129 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=114.05 KB/116790 for 5dcf6997c5a081edf1d95a6a373c6b08 in 524ms, sequenceid=14, compaction requested=false 2023-04-16 00:54:46,129 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. started... 2023-04-16 00:54:46,129 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 5dcf6997c5a081edf1d95a6a373c6b08: 2023-04-16 00:54:46,129 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 5dcf6997c5a081edf1d95a6a373c6b08 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-16 00:54:46,129 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5dcf6997c5a081edf1d95a6a373c6b08, store=A 2023-04-16 00:54:46,129 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:54:46,129 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5dcf6997c5a081edf1d95a6a373c6b08, store=B 2023-04-16 00:54:46,129 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:54:46,129 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5dcf6997c5a081edf1d95a6a373c6b08, store=C 2023-04-16 00:54:46,129 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:54:46,241 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(9158): Flush requested on 5dcf6997c5a081edf1d95a6a373c6b08 2023-04-16 00:54:46,241 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. as already flushing 2023-04-16 00:54:46,248 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:46,248 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58306 deadline: 1681606546245, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:46,248 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:46,248 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58336 deadline: 1681606546245, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:46,248 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:46,248 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58292 deadline: 1681606546246, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:46,250 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:46,250 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58322 deadline: 1681606546248, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:46,250 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:46,250 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58282 deadline: 1681606546248, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:46,350 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:46,350 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:46,350 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58306 deadline: 1681606546349, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:46,350 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58292 deadline: 1681606546349, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:46,350 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:46,350 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58336 deadline: 1681606546349, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:46,352 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:46,352 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:46,352 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58322 deadline: 1681606546351, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:46,352 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58282 deadline: 1681606546351, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:46,539 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=34 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/.tmp/A/763d2b70147c46289187c95232169c9e 2023-04-16 00:54:46,553 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:46,553 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58306 deadline: 1681606546552, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:46,553 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:46,553 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:46,553 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58292 deadline: 1681606546552, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:46,553 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58336 deadline: 1681606546552, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:46,553 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:46,553 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58282 deadline: 1681606546553, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:46,554 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:46,554 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58322 deadline: 1681606546553, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:46,562 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=34 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/.tmp/B/261b40f4e6f44095baa2081690ff7c51 2023-04-16 00:54:46,856 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:46,856 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58322 deadline: 1681606546854, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:46,856 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:46,857 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58336 deadline: 1681606546855, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:46,857 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:46,857 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58282 deadline: 1681606546855, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:46,857 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:46,857 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58306 deadline: 1681606546856, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:46,857 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:46,857 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58292 deadline: 1681606546856, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:46,972 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=34 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/.tmp/C/bc37fce4d91344a199d34a2b4d21ecec 2023-04-16 00:54:46,976 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/.tmp/A/763d2b70147c46289187c95232169c9e as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/A/763d2b70147c46289187c95232169c9e 2023-04-16 00:54:46,979 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/A/763d2b70147c46289187c95232169c9e, entries=150, sequenceid=34, filesize=11.6 K 2023-04-16 00:54:46,980 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/.tmp/B/261b40f4e6f44095baa2081690ff7c51 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/B/261b40f4e6f44095baa2081690ff7c51 2023-04-16 00:54:46,983 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/B/261b40f4e6f44095baa2081690ff7c51, entries=150, sequenceid=34, filesize=11.6 K 2023-04-16 00:54:46,984 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/.tmp/C/bc37fce4d91344a199d34a2b4d21ecec as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/C/bc37fce4d91344a199d34a2b4d21ecec 2023-04-16 00:54:46,988 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/C/bc37fce4d91344a199d34a2b4d21ecec, entries=150, sequenceid=34, filesize=11.6 K 2023-04-16 00:54:46,988 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=67.09 KB/68700 for 5dcf6997c5a081edf1d95a6a373c6b08 in 859ms, sequenceid=34, compaction requested=false 2023-04-16 00:54:46,988 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 5dcf6997c5a081edf1d95a6a373c6b08: 2023-04-16 00:54:46,988 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. 2023-04-16 00:54:46,988 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:54:46,988 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:54:46,989 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:46,989 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:54:46,989 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:54:46,990 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:46,990 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:46,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:46,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:46,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:46,990 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:46,990 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:54:46,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:46,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:46,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:46,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:46,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:46,992 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:54:46,992 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@59e108ff[Count = 0] remaining members to acquire global barrier 2023-04-16 00:54:46,992 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:54:46,992 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:46,993 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:46,993 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:46,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:46,993 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:54:46,993 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:54:46,993 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:54:46,994 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:46,994 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:54:46,995 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:54:46,995 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:54:46,995 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:46,995 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:54:46,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:46,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:46,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:46,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:46,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:46,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:46,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:46,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:46,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:46,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:46,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:54:46,997 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:54:46,998 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:54:46,998 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:54:46,998 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:54:46,998 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:54:46,998 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:54:47,000 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:47,000 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:47,000 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:47,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:47,000 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:54:47,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:47,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:47,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:47,001 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:47,001 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:54:47,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:54:47,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:47,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:47,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:47,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:47,002 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:47,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:47,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:47,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:47,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:47,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:47,005 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:47,005 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:47,005 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:47,005 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:54:47,005 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:54:47,006 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:54:47,006 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:54:47,006 DEBUG [Thread-1736] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:54:47,005 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:47,006 DEBUG [Thread-1736] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:54:47,006 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:47,006 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:54:47,006 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:54:47,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:54:47,006 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:54:47,006 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:47,006 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:47,007 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:54:47,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:54:47,106 DEBUG [Thread-1736] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:54:47,107 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:54:47,108 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:54:47,110 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:54:47,110 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:54:47,110 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:54:47,110 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:54:47,110 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:54:47,110 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:54:47,110 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:47,110 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:47,111 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:54:47,111 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:47,111 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:54:47,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:54:47,112 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:47,112 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:54:47,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:47,112 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:47,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:54:47,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:47,113 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:54:47,113 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:54:47,113 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:54:47,113 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:54:47,113 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:54:47,113 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:54:47,113 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:54:47,113 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:54:47,113 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. 2023-04-16 00:54:47,113 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. started... 2023-04-16 00:54:47,114 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 5dcf6997c5a081edf1d95a6a373c6b08 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-16 00:54:47,114 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5dcf6997c5a081edf1d95a6a373c6b08, store=A 2023-04-16 00:54:47,114 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:54:47,114 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5dcf6997c5a081edf1d95a6a373c6b08, store=B 2023-04-16 00:54:47,114 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:54:47,114 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5dcf6997c5a081edf1d95a6a373c6b08, store=C 2023-04-16 00:54:47,114 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:54:47,359 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(9158): Flush requested on 5dcf6997c5a081edf1d95a6a373c6b08 2023-04-16 00:54:47,359 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. as already flushing 2023-04-16 00:54:47,373 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:47,373 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58336 deadline: 1681606547370, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:47,375 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:47,375 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58292 deadline: 1681606547372, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:47,375 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:47,375 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58306 deadline: 1681606547372, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:47,375 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:47,376 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58282 deadline: 1681606547373, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:47,376 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:47,376 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58322 deadline: 1681606547373, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:47,476 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:47,476 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58336 deadline: 1681606547474, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:47,478 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:47,478 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58306 deadline: 1681606547476, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:47,478 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:47,478 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58282 deadline: 1681606547476, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:47,478 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:47,478 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58322 deadline: 1681606547477, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:47,559 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=47 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/.tmp/A/455293189bd74ec48814096fbe126c40 2023-04-16 00:54:47,679 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:47,680 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58336 deadline: 1681606547677, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:47,682 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:47,682 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58306 deadline: 1681606547679, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:47,682 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:47,682 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58322 deadline: 1681606547679, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:47,683 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:47,683 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58282 deadline: 1681606547680, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:47,970 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=47 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/.tmp/B/d66b1d97395345d0a463b6b3ed8ebe0a 2023-04-16 00:54:47,980 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=47 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/.tmp/C/45b7fbb66e384f82be01c400fa6913f0 2023-04-16 00:54:47,982 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:47,983 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58336 deadline: 1681606547982, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:47,983 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:47,984 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58306 deadline: 1681606547983, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:47,984 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/.tmp/A/455293189bd74ec48814096fbe126c40 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/A/455293189bd74ec48814096fbe126c40 2023-04-16 00:54:47,985 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:47,985 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58322 deadline: 1681606547983, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:47,987 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:47,987 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58282 deadline: 1681606547986, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:47,989 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/A/455293189bd74ec48814096fbe126c40, entries=150, sequenceid=47, filesize=11.6 K 2023-04-16 00:54:47,990 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/.tmp/B/d66b1d97395345d0a463b6b3ed8ebe0a as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/B/d66b1d97395345d0a463b6b3ed8ebe0a 2023-04-16 00:54:47,998 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/B/d66b1d97395345d0a463b6b3ed8ebe0a, entries=150, sequenceid=47, filesize=11.6 K 2023-04-16 00:54:47,999 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/.tmp/C/45b7fbb66e384f82be01c400fa6913f0 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/C/45b7fbb66e384f82be01c400fa6913f0 2023-04-16 00:54:48,002 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/C/45b7fbb66e384f82be01c400fa6913f0, entries=150, sequenceid=47, filesize=11.6 K 2023-04-16 00:54:48,003 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=114.05 KB/116790 for 5dcf6997c5a081edf1d95a6a373c6b08 in 889ms, sequenceid=47, compaction requested=true 2023-04-16 00:54:48,003 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 5dcf6997c5a081edf1d95a6a373c6b08: 2023-04-16 00:54:48,003 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. 2023-04-16 00:54:48,003 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:54:48,003 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:54:48,003 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:48,003 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:54:48,003 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:54:48,004 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:48,004 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:48,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:48,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:48,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:48,005 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:48,005 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:54:48,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:48,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:48,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:48,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:48,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:48,006 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:54:48,006 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@68e708cf[Count = 0] remaining members to acquire global barrier 2023-04-16 00:54:48,006 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:54:48,007 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:48,008 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:48,009 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:48,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:48,009 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:54:48,009 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:54:48,009 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:54:48,009 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:48,009 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:54:48,010 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:54:48,010 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:54:48,010 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:48,010 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:54:48,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:48,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:48,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:48,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:48,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:48,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:48,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:48,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:48,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:48,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:48,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:54:48,013 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:54:48,013 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:54:48,013 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:54:48,013 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:54:48,013 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:54:48,013 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:54:48,014 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:48,014 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:48,014 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:48,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:48,014 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:54:48,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:48,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:48,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:48,015 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:54:48,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:54:48,015 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:48,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:48,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:48,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:48,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:48,016 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:48,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:48,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:48,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:48,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:48,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:48,018 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:54:48,018 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:48,018 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:54:48,018 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:54:48,018 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:54:48,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:54:48,019 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:54:48,019 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:54:48,018 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:48,019 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:48,019 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:48,019 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:48,019 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:48,019 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:54:48,019 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:54:48,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:54:48,019 DEBUG [Thread-1736] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:54:48,019 DEBUG [Thread-1736] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:54:48,019 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:48,119 DEBUG [Thread-1736] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:54:48,120 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:54:48,121 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:54:48,122 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:54:48,122 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:54:48,122 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:54:48,123 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:54:48,123 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:54:48,123 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:54:48,123 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:48,123 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:48,125 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:54:48,125 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:48,125 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:54:48,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:54:48,126 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:48,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:48,126 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:54:48,126 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:48,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:54:48,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:48,126 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:54:48,127 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:54:48,127 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:54:48,127 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:54:48,127 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:54:48,127 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:54:48,127 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:54:48,127 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:54:48,127 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. 2023-04-16 00:54:48,127 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. started... 2023-04-16 00:54:48,127 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 5dcf6997c5a081edf1d95a6a373c6b08 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-16 00:54:48,127 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5dcf6997c5a081edf1d95a6a373c6b08, store=A 2023-04-16 00:54:48,128 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:54:48,128 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5dcf6997c5a081edf1d95a6a373c6b08, store=B 2023-04-16 00:54:48,128 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:54:48,128 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5dcf6997c5a081edf1d95a6a373c6b08, store=C 2023-04-16 00:54:48,128 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:54:48,135 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=67 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/.tmp/A/3bcea7d9776f4f0b81265429ea0ef32f 2023-04-16 00:54:48,379 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(9158): Flush requested on 5dcf6997c5a081edf1d95a6a373c6b08 2023-04-16 00:54:48,379 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. as already flushing 2023-04-16 00:54:48,403 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:48,403 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58292 deadline: 1681606548400, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:48,486 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:48,487 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58306 deadline: 1681606548485, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:48,487 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:48,487 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58336 deadline: 1681606548485, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:48,489 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:48,489 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58322 deadline: 1681606548488, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:48,490 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:48,490 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58282 deadline: 1681606548489, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:48,505 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:48,506 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58292 deadline: 1681606548504, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:48,546 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=67 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/.tmp/B/0f1483715d8c4ee096ca03b10a1ccd11 2023-04-16 00:54:48,618 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-16 00:54:48,708 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:48,709 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58292 deadline: 1681606548706, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:48,956 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=67 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/.tmp/C/5ad005f3206149d9827cf860b162acd9 2023-04-16 00:54:48,966 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/.tmp/A/3bcea7d9776f4f0b81265429ea0ef32f as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/A/3bcea7d9776f4f0b81265429ea0ef32f 2023-04-16 00:54:48,970 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/A/3bcea7d9776f4f0b81265429ea0ef32f, entries=150, sequenceid=67, filesize=11.6 K 2023-04-16 00:54:48,971 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/.tmp/B/0f1483715d8c4ee096ca03b10a1ccd11 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/B/0f1483715d8c4ee096ca03b10a1ccd11 2023-04-16 00:54:48,981 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/B/0f1483715d8c4ee096ca03b10a1ccd11, entries=150, sequenceid=67, filesize=11.6 K 2023-04-16 00:54:48,982 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/.tmp/C/5ad005f3206149d9827cf860b162acd9 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/C/5ad005f3206149d9827cf860b162acd9 2023-04-16 00:54:48,986 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/C/5ad005f3206149d9827cf860b162acd9, entries=150, sequenceid=67, filesize=11.6 K 2023-04-16 00:54:48,986 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for 5dcf6997c5a081edf1d95a6a373c6b08 in 859ms, sequenceid=67, compaction requested=true 2023-04-16 00:54:48,986 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 5dcf6997c5a081edf1d95a6a373c6b08: 2023-04-16 00:54:48,986 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. 2023-04-16 00:54:48,987 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:54:48,987 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:54:48,987 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:48,987 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:54:48,987 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:54:48,988 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:48,988 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:48,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:48,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:48,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:48,989 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:48,989 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:54:48,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:48,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:48,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:48,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:48,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:48,990 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:54:48,990 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5f81a54c[Count = 0] remaining members to acquire global barrier 2023-04-16 00:54:48,990 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:54:48,990 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:48,991 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:48,991 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:48,991 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:54:48,991 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:48,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:48,992 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:54:48,992 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:54:48,992 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:54:48,993 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:54:48,993 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:48,993 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:54:48,993 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:54:48,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:48,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:48,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:48,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:48,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:48,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:48,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:48,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:48,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:48,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:48,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:54:48,996 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:54:48,996 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:54:48,996 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:54:48,996 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:54:48,996 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:54:48,996 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:54:48,998 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:48,998 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:48,998 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:54:48,998 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:48,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:48,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:48,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:48,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:48,999 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:54:48,999 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:48,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:48,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:54:48,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:49,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:49,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:49,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:49,000 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:49,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:49,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:49,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:49,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:49,003 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:49,003 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:49,003 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:49,003 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:49,003 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:54:49,003 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:54:49,003 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:54:49,003 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:54:49,003 DEBUG [Thread-1736] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:54:49,003 DEBUG [Thread-1736] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:54:49,003 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:54:49,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:54:49,003 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:54:49,003 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:49,004 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:49,004 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:49,004 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:54:49,004 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:54:49,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:54:49,013 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(9158): Flush requested on 5dcf6997c5a081edf1d95a6a373c6b08 2023-04-16 00:54:49,013 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 5dcf6997c5a081edf1d95a6a373c6b08 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-16 00:54:49,014 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5dcf6997c5a081edf1d95a6a373c6b08, store=A 2023-04-16 00:54:49,014 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:54:49,014 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5dcf6997c5a081edf1d95a6a373c6b08, store=B 2023-04-16 00:54:49,014 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:54:49,014 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5dcf6997c5a081edf1d95a6a373c6b08, store=C 2023-04-16 00:54:49,014 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:54:49,031 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=80 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/.tmp/A/5da8050d3e274b71a9e194c9cb082c26 2023-04-16 00:54:49,041 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=80 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/.tmp/B/cb9ce6a037de40c282327f598b27e035 2023-04-16 00:54:49,080 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:49,080 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58292 deadline: 1681606549076, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:49,104 DEBUG [Thread-1736] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:54:49,104 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:54:49,106 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:54:49,108 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:54:49,108 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:54:49,108 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:54:49,108 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:54:49,108 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:54:49,108 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:54:49,108 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:49,108 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:49,112 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:49,112 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:54:49,112 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:54:49,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:54:49,112 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:49,112 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:54:49,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:49,113 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:49,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:54:49,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:49,113 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:54:49,113 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:54:49,113 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:54:49,114 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:54:49,114 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:54:49,114 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:54:49,114 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:54:49,114 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:54:49,114 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. 2023-04-16 00:54:49,114 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. started... 2023-04-16 00:54:49,114 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. as already flushing 2023-04-16 00:54:49,114 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. 2023-04-16 00:54:49,114 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:54:49,114 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:54:49,114 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:49,114 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:54:49,114 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:54:49,119 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:49,119 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:49,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:49,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:49,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:49,120 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:49,120 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:54:49,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:49,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:49,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:49,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:49,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:49,121 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:54:49,121 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@398ee744[Count = 0] remaining members to acquire global barrier 2023-04-16 00:54:49,121 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:54:49,121 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:49,122 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:49,122 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:49,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:49,122 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:54:49,122 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:54:49,122 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:54:49,123 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:49,123 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:54:49,124 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:49,124 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:54:49,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:49,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:49,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:49,124 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:54:49,124 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:54:49,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:49,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:49,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:49,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:49,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:49,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:49,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:49,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:54:49,126 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:54:49,126 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:54:49,126 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:54:49,126 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:54:49,126 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:54:49,126 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:54:49,128 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:49,128 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:49,128 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:49,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:49,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:49,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:49,128 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:54:49,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:49,128 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:49,128 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:54:49,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:54:49,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:49,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:49,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:49,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:49,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:49,130 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:49,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:49,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:49,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:49,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:49,133 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:49,133 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:49,133 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:54:49,133 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:54:49,133 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:54:49,133 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:54:49,133 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:54:49,133 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:54:49,133 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:49,133 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:54:49,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:54:49,133 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:49,133 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:49,133 DEBUG [Thread-1736] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:54:49,134 DEBUG [Thread-1736] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:54:49,133 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:49,134 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:49,134 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:54:49,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:54:49,184 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:49,184 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58292 deadline: 1681606549181, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:49,234 DEBUG [Thread-1736] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:54:49,234 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:54:49,236 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:54:49,237 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:54:49,237 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:54:49,237 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:54:49,237 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:54:49,237 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:54:49,237 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:54:49,237 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:49,238 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:49,239 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:54:49,239 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:54:49,239 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:49,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:54:49,239 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:49,239 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:54:49,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:49,240 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:49,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:54:49,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:49,240 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:54:49,240 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:54:49,240 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:54:49,240 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:54:49,241 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:54:49,241 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:54:49,241 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:54:49,241 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:54:49,241 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. 2023-04-16 00:54:49,241 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. started... 2023-04-16 00:54:49,241 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. as already flushing 2023-04-16 00:54:49,241 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. 2023-04-16 00:54:49,241 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:54:49,241 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:54:49,241 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:49,241 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:54:49,241 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:54:49,242 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:49,242 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:49,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:49,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:49,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:49,243 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:49,243 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:54:49,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:49,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:49,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:49,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:49,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:49,244 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:54:49,244 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@68f9a67a[Count = 0] remaining members to acquire global barrier 2023-04-16 00:54:49,244 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:54:49,244 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:49,245 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:49,245 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:49,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:49,245 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:49,245 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:54:49,245 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:54:49,245 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:54:49,245 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:54:49,248 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:49,248 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:54:49,248 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:54:49,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:49,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:49,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:49,248 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:54:49,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:49,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:49,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:49,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:49,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:49,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:49,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:49,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:54:49,250 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:54:49,250 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:54:49,250 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:54:49,250 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:54:49,250 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:54:49,250 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:54:49,251 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:49,251 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:49,252 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:54:49,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:49,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:49,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:49,252 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:49,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:49,252 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:49,252 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:54:49,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:54:49,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:49,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:49,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:49,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:49,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:49,253 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:49,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:49,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:49,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:49,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:49,256 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:54:49,256 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:49,256 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:54:49,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:54:49,256 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:54:49,256 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:49,256 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:49,256 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:54:49,257 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:54:49,256 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:49,257 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:49,257 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:49,257 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:49,256 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:54:49,257 DEBUG [Thread-1736] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:54:49,257 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:54:49,257 DEBUG [Thread-1736] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:54:49,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:54:49,263 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:54:49,357 DEBUG [Thread-1736] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:54:49,358 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:54:49,359 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:54:49,360 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:54:49,360 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:54:49,360 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:54:49,361 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:54:49,361 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:54:49,361 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:54:49,361 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:49,361 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:49,362 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:49,362 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:54:49,362 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:54:49,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:54:49,363 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:49,363 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:54:49,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:49,363 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:49,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:54:49,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:49,363 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:54:49,364 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:54:49,364 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:54:49,364 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:54:49,364 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:54:49,364 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:54:49,365 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:54:49,365 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:54:49,365 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. 2023-04-16 00:54:49,365 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. started... 2023-04-16 00:54:49,365 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. as already flushing 2023-04-16 00:54:49,365 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. 2023-04-16 00:54:49,365 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:54:49,365 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:54:49,365 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:49,365 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:54:49,365 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:54:49,367 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:49,367 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:49,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:49,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:49,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:49,368 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:49,368 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:54:49,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:49,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:49,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:49,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:49,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:49,369 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:54:49,369 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@38087ada[Count = 0] remaining members to acquire global barrier 2023-04-16 00:54:49,369 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:54:49,369 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:49,370 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:49,370 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:49,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:49,370 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:54:49,370 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:54:49,370 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:49,371 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:54:49,371 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:54:49,372 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:49,372 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:54:49,372 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:54:49,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:49,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:49,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:49,372 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:54:49,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:49,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:49,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:49,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:49,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:49,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:49,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:49,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:54:49,374 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:54:49,374 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:54:49,374 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:54:49,374 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:54:49,374 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:54:49,375 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:54:49,376 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:49,376 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:54:49,376 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:49,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:49,376 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:49,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:49,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:49,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:49,376 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:54:49,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:54:49,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:49,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:49,377 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:49,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:49,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:49,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:49,378 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:49,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:49,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:49,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:49,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:49,381 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:54:49,381 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:54:49,381 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:54:49,381 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:54:49,381 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:54:49,381 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:49,381 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:54:49,381 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:54:49,382 DEBUG [Thread-1736] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:54:49,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:54:49,381 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:49,382 DEBUG [Thread-1736] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:54:49,382 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:49,382 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:49,382 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:49,382 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:54:49,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:54:49,382 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:49,382 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:49,386 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:49,387 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58292 deadline: 1681606549385, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:49,451 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=80 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/.tmp/C/edbcee3cbb9c467aa1bc5a2d92765b97 2023-04-16 00:54:49,455 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/.tmp/A/5da8050d3e274b71a9e194c9cb082c26 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/A/5da8050d3e274b71a9e194c9cb082c26 2023-04-16 00:54:49,458 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/A/5da8050d3e274b71a9e194c9cb082c26, entries=200, sequenceid=80, filesize=13.9 K 2023-04-16 00:54:49,459 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/.tmp/B/cb9ce6a037de40c282327f598b27e035 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/B/cb9ce6a037de40c282327f598b27e035 2023-04-16 00:54:49,462 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/B/cb9ce6a037de40c282327f598b27e035, entries=150, sequenceid=80, filesize=11.6 K 2023-04-16 00:54:49,465 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/.tmp/C/edbcee3cbb9c467aa1bc5a2d92765b97 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/C/edbcee3cbb9c467aa1bc5a2d92765b97 2023-04-16 00:54:49,471 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/C/edbcee3cbb9c467aa1bc5a2d92765b97, entries=150, sequenceid=80, filesize=11.6 K 2023-04-16 00:54:49,475 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for 5dcf6997c5a081edf1d95a6a373c6b08 in 462ms, sequenceid=80, compaction requested=true 2023-04-16 00:54:49,475 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 5dcf6997c5a081edf1d95a6a373c6b08: 2023-04-16 00:54:49,475 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 00:54:49,475 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-16 00:54:49,475 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-16 00:54:49,475 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 00:54:49,475 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 00:54:49,478 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 64110 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-16 00:54:49,478 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HStore(1912): 5dcf6997c5a081edf1d95a6a373c6b08/A is initiating minor compaction (all files) 2023-04-16 00:54:49,478 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 5dcf6997c5a081edf1d95a6a373c6b08/A in TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. 2023-04-16 00:54:49,478 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/A/234b83427e2c4b31b199e2d41ca9a8b7, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/A/763d2b70147c46289187c95232169c9e, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/A/455293189bd74ec48814096fbe126c40, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/A/3bcea7d9776f4f0b81265429ea0ef32f, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/A/5da8050d3e274b71a9e194c9cb082c26] into tmpdir=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/.tmp, totalSize=62.6 K 2023-04-16 00:54:49,478 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 59430 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-16 00:54:49,478 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.Compactor(207): Compacting 234b83427e2c4b31b199e2d41ca9a8b7, keycount=200, bloomtype=ROW, size=13.9 K, encoding=NONE, compression=NONE, seqNum=14, earliestPutTs=1681606485595 2023-04-16 00:54:49,479 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HStore(1912): 5dcf6997c5a081edf1d95a6a373c6b08/B is initiating minor compaction (all files) 2023-04-16 00:54:49,479 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 5dcf6997c5a081edf1d95a6a373c6b08/B in TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. 2023-04-16 00:54:49,479 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/B/1a1b8c00fba64695895917834d3a77c5, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/B/261b40f4e6f44095baa2081690ff7c51, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/B/d66b1d97395345d0a463b6b3ed8ebe0a, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/B/0f1483715d8c4ee096ca03b10a1ccd11, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/B/cb9ce6a037de40c282327f598b27e035] into tmpdir=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/.tmp, totalSize=58.0 K 2023-04-16 00:54:49,479 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.Compactor(207): Compacting 763d2b70147c46289187c95232169c9e, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=34, earliestPutTs=1681606485624 2023-04-16 00:54:49,479 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting 1a1b8c00fba64695895917834d3a77c5, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=14, earliestPutTs=1681606485595 2023-04-16 00:54:49,479 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.Compactor(207): Compacting 455293189bd74ec48814096fbe126c40, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=47, earliestPutTs=1681606486243 2023-04-16 00:54:49,479 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting 261b40f4e6f44095baa2081690ff7c51, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=34, earliestPutTs=1681606485624 2023-04-16 00:54:49,480 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.Compactor(207): Compacting 3bcea7d9776f4f0b81265429ea0ef32f, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=67, earliestPutTs=1681606487368 2023-04-16 00:54:49,480 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting d66b1d97395345d0a463b6b3ed8ebe0a, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=47, earliestPutTs=1681606486243 2023-04-16 00:54:49,480 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.Compactor(207): Compacting 5da8050d3e274b71a9e194c9cb082c26, keycount=200, bloomtype=ROW, size=13.9 K, encoding=NONE, compression=NONE, seqNum=80, earliestPutTs=1681606488388 2023-04-16 00:54:49,480 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting 0f1483715d8c4ee096ca03b10a1ccd11, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=67, earliestPutTs=1681606487368 2023-04-16 00:54:49,481 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting cb9ce6a037de40c282327f598b27e035, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=80, earliestPutTs=1681606488396 2023-04-16 00:54:49,482 DEBUG [Thread-1736] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:54:49,482 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:54:49,484 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:54:49,486 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:54:49,486 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:54:49,486 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:54:49,486 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:54:49,486 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:54:49,486 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:54:49,488 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:49,488 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:49,489 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:49,489 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:54:49,489 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:54:49,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:54:49,489 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:49,490 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:54:49,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:49,490 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:49,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:54:49,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:49,490 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:54:49,491 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:54:49,491 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:54:49,491 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:54:49,491 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:54:49,491 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:54:49,494 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(9158): Flush requested on 5dcf6997c5a081edf1d95a6a373c6b08 2023-04-16 00:54:49,495 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 5dcf6997c5a081edf1d95a6a373c6b08 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-16 00:54:49,495 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5dcf6997c5a081edf1d95a6a373c6b08, store=A 2023-04-16 00:54:49,495 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:54:49,495 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5dcf6997c5a081edf1d95a6a373c6b08, store=B 2023-04-16 00:54:49,495 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:54:49,495 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5dcf6997c5a081edf1d95a6a373c6b08, store=C 2023-04-16 00:54:49,495 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:54:49,499 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:54:49,499 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:54:49,499 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. 2023-04-16 00:54:49,500 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. started... 2023-04-16 00:54:49,500 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. as already flushing 2023-04-16 00:54:49,500 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. 2023-04-16 00:54:49,500 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:54:49,500 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:54:49,500 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:49,500 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:54:49,500 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:54:49,502 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] throttle.PressureAwareThroughputController(145): 5dcf6997c5a081edf1d95a6a373c6b08#A#compaction#233 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-16 00:54:49,502 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:49,502 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:49,502 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:54:49,502 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] throttle.PressureAwareThroughputController(145): 5dcf6997c5a081edf1d95a6a373c6b08#B#compaction#232 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-16 00:54:49,503 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:49,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:49,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:49,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:49,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:49,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:49,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:49,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:49,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:49,507 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:54:49,508 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6a711352[Count = 0] remaining members to acquire global barrier 2023-04-16 00:54:49,508 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:54:49,508 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:49,511 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:49,511 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:49,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:49,511 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:54:49,513 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:49,513 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:54:49,513 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:54:49,513 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:54:49,514 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:49,514 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:54:49,515 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:54:49,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:49,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:49,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:49,515 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:54:49,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:49,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:49,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:49,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:49,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:49,517 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:49,517 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58306 deadline: 1681606549510, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:49,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:49,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:49,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:54:49,518 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:54:49,518 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:54:49,518 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:54:49,518 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:54:49,519 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:54:49,519 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:54:49,519 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/.tmp/B/1699ebb87e4844ca9710c20f85a137ee as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/B/1699ebb87e4844ca9710c20f85a137ee 2023-04-16 00:54:49,520 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:49,521 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:54:49,521 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:49,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:49,521 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:49,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:49,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:49,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:49,521 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:54:49,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:54:49,521 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:49,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:49,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:49,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:49,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:49,522 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:49,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:49,522 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:49,522 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58336 deadline: 1681606549517, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:49,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:49,523 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:49,523 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58282 deadline: 1681606549517, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:49,523 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:49,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:49,523 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 5dcf6997c5a081edf1d95a6a373c6b08/B of 5dcf6997c5a081edf1d95a6a373c6b08 into 1699ebb87e4844ca9710c20f85a137ee(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 00:54:49,523 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58322 deadline: 1681606549519, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:49,523 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5dcf6997c5a081edf1d95a6a373c6b08: 2023-04-16 00:54:49,523 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08., storeName=5dcf6997c5a081edf1d95a6a373c6b08/B, priority=11, startTime=1681606489475; duration=0sec 2023-04-16 00:54:49,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:49,524 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 00:54:49,524 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-16 00:54:49,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:49,525 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:49,525 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:49,525 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:49,525 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:49,525 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:54:49,525 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:54:49,525 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:54:49,525 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:54:49,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:54:49,525 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 59430 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-16 00:54:49,525 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HStore(1912): 5dcf6997c5a081edf1d95a6a373c6b08/C is initiating minor compaction (all files) 2023-04-16 00:54:49,525 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:49,526 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:49,525 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 5dcf6997c5a081edf1d95a6a373c6b08/C in TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. 2023-04-16 00:54:49,526 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/C/90cf6ae6569f4bc6a55737127551399f, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/C/bc37fce4d91344a199d34a2b4d21ecec, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/C/45b7fbb66e384f82be01c400fa6913f0, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/C/5ad005f3206149d9827cf860b162acd9, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/C/edbcee3cbb9c467aa1bc5a2d92765b97] into tmpdir=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/.tmp, totalSize=58.0 K 2023-04-16 00:54:49,525 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:54:49,526 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:54:49,525 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:54:49,526 DEBUG [Thread-1736] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:54:49,526 DEBUG [Thread-1736] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:54:49,526 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:54:49,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:54:49,526 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:49,526 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting 90cf6ae6569f4bc6a55737127551399f, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=14, earliestPutTs=1681606485595 2023-04-16 00:54:49,527 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting bc37fce4d91344a199d34a2b4d21ecec, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=34, earliestPutTs=1681606485624 2023-04-16 00:54:49,527 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting 45b7fbb66e384f82be01c400fa6913f0, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=47, earliestPutTs=1681606486243 2023-04-16 00:54:49,527 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting 5ad005f3206149d9827cf860b162acd9, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=67, earliestPutTs=1681606487368 2023-04-16 00:54:49,527 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting edbcee3cbb9c467aa1bc5a2d92765b97, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=80, earliestPutTs=1681606488396 2023-04-16 00:54:49,535 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] throttle.PressureAwareThroughputController(145): 5dcf6997c5a081edf1d95a6a373c6b08#C#compaction#234 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-16 00:54:49,619 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:49,619 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58306 deadline: 1681606549618, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:49,624 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:49,624 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58336 deadline: 1681606549623, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:49,625 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:49,626 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58282 deadline: 1681606549624, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:49,626 DEBUG [Thread-1736] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:54:49,627 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:49,627 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:54:49,627 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58322 deadline: 1681606549624, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:49,628 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:54:49,629 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:54:49,629 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:54:49,629 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:54:49,629 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:54:49,629 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:54:49,629 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:54:49,630 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:49,630 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:49,632 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:49,632 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:54:49,632 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:54:49,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:54:49,632 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:49,632 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:54:49,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:49,633 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:49,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:54:49,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:49,633 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:54:49,633 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:54:49,633 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:54:49,633 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:54:49,634 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:54:49,634 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:54:49,634 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:54:49,634 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:54:49,634 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. 2023-04-16 00:54:49,634 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. started... 2023-04-16 00:54:49,634 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. as already flushing 2023-04-16 00:54:49,634 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. 2023-04-16 00:54:49,634 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:54:49,634 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:54:49,634 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:49,634 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:54:49,634 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:54:49,636 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:49,636 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:49,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:49,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:49,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:49,636 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:49,636 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:54:49,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:49,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:49,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:49,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:49,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:49,637 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:54:49,638 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@240b7d9f[Count = 0] remaining members to acquire global barrier 2023-04-16 00:54:49,638 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:54:49,638 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:49,639 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:49,639 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:49,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:49,639 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:54:49,639 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:54:49,639 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:54:49,639 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:49,639 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:54:49,640 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:49,640 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:54:49,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:49,640 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:54:49,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:49,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:49,640 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:54:49,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:49,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:49,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:49,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:49,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:49,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:49,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:49,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:54:49,643 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:54:49,643 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:54:49,643 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:54:49,643 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:54:49,643 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:54:49,643 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:54:49,645 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:49,645 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:49,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:49,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:49,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:49,646 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:49,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:49,646 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:54:49,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:49,646 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:54:49,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:54:49,646 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:49,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:49,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:49,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:49,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:49,648 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:49,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:49,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:49,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:49,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:49,650 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:49,650 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:54:49,650 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:49,650 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:54:49,650 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:49,650 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:54:49,650 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:54:49,650 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:54:49,650 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:54:49,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:54:49,650 DEBUG [Thread-1736] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:54:49,650 DEBUG [Thread-1736] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:54:49,650 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:49,650 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:49,650 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:49,650 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:54:49,650 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:49,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:54:49,691 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:49,691 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58292 deadline: 1681606549689, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:49,750 DEBUG [Thread-1736] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:54:49,751 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:54:49,752 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:54:49,753 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:54:49,754 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:54:49,754 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:54:49,754 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:54:49,754 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:54:49,754 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:54:49,754 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:49,754 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:49,755 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:49,755 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:54:49,755 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:54:49,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:54:49,756 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:49,756 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:54:49,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:49,756 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:49,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:54:49,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:49,757 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:54:49,757 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:54:49,757 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:54:49,757 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:54:49,757 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:54:49,757 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:54:49,757 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:54:49,757 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. 2023-04-16 00:54:49,757 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. started... 2023-04-16 00:54:49,757 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:54:49,757 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. as already flushing 2023-04-16 00:54:49,757 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. 2023-04-16 00:54:49,757 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:54:49,757 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:54:49,757 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:49,757 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:54:49,757 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:54:49,759 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:49,759 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:49,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:49,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:49,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:49,759 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:49,759 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:54:49,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:49,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:49,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:49,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:49,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:49,761 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:54:49,761 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@164c512[Count = 0] remaining members to acquire global barrier 2023-04-16 00:54:49,761 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:54:49,761 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:49,763 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:49,763 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:49,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:49,763 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:54:49,763 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:54:49,763 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:54:49,763 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:49,763 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:54:49,764 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:54:49,764 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:49,764 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:54:49,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:49,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:49,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:49,764 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:54:49,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:49,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:49,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:49,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:49,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:49,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:49,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:49,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:54:49,766 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:54:49,766 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:54:49,767 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:54:49,767 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:54:49,767 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:54:49,767 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:54:49,768 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:49,768 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:49,768 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:49,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:49,768 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:54:49,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:49,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:49,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:49,769 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:49,769 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:54:49,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:54:49,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:49,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:49,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:49,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:49,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:49,770 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:49,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:49,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:49,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:49,772 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:54:49,772 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:49,772 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:54:49,772 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:54:49,772 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:54:49,772 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:54:49,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:54:49,772 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:54:49,772 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:49,772 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:54:49,772 DEBUG [Thread-1736] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:54:49,772 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:49,772 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:54:49,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:54:49,772 DEBUG [Thread-1736] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:54:49,772 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:49,773 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:49,773 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:49,773 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:49,822 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:49,822 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58306 deadline: 1681606549821, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:49,827 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:49,827 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58336 deadline: 1681606549826, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:49,829 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:49,829 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58282 deadline: 1681606549827, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:49,829 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:49,829 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58322 deadline: 1681606549828, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:49,873 DEBUG [Thread-1736] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:54:49,873 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:54:49,875 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:54:49,876 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:54:49,876 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:54:49,876 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:54:49,876 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:54:49,876 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:54:49,876 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:54:49,876 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:49,876 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:49,879 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:49,879 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:54:49,879 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:54:49,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:54:49,879 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:49,879 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:54:49,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:49,880 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:49,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:54:49,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:49,880 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:54:49,880 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:54:49,880 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:54:49,880 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:54:49,880 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:54:49,880 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:54:49,881 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:54:49,881 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:54:49,881 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. 2023-04-16 00:54:49,881 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. started... 2023-04-16 00:54:49,881 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. as already flushing 2023-04-16 00:54:49,881 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. 2023-04-16 00:54:49,881 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:54:49,881 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:54:49,881 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:49,881 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:54:49,881 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:54:49,882 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:49,882 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:49,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:49,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:49,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:49,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:49,883 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:49,883 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:54:49,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:49,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:49,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:49,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:49,884 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:54:49,884 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7ac1a14e[Count = 0] remaining members to acquire global barrier 2023-04-16 00:54:49,884 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:54:49,884 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:49,885 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:49,885 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:49,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:49,886 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:54:49,886 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:54:49,886 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:49,886 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:54:49,886 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:54:49,894 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:54:49,894 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:49,894 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:54:49,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:49,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:49,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:49,894 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:54:49,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:49,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:49,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:49,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:49,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:49,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:49,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:49,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:54:49,896 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:54:49,896 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:54:49,896 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:54:49,896 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:54:49,897 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:54:49,897 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:54:49,898 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:49,898 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:49,898 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:49,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:49,898 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:54:49,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:49,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:49,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:49,898 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:54:49,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:49,899 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:49,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:54:49,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:49,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:49,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:49,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:49,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:49,900 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:49,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:49,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:49,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:49,902 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:49,902 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:49,902 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:49,902 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:54:49,902 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:49,903 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:54:49,903 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:54:49,903 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:54:49,903 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:54:49,903 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:54:49,903 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:54:49,903 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:49,903 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:49,903 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:49,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:54:49,903 DEBUG [Thread-1736] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:54:49,903 DEBUG [Thread-1736] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:54:49,903 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:54:49,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:54:49,911 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=100 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/.tmp/A/ddfda75fb7554390b8b938dca019fdaa 2023-04-16 00:54:49,924 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/.tmp/A/d100f6235c09495fa58b0f7fa41a0a28 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/A/d100f6235c09495fa58b0f7fa41a0a28 2023-04-16 00:54:49,929 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 5dcf6997c5a081edf1d95a6a373c6b08/A of 5dcf6997c5a081edf1d95a6a373c6b08 into d100f6235c09495fa58b0f7fa41a0a28(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 00:54:49,929 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5dcf6997c5a081edf1d95a6a373c6b08: 2023-04-16 00:54:49,929 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08., storeName=5dcf6997c5a081edf1d95a6a373c6b08/A, priority=11, startTime=1681606489475; duration=0sec 2023-04-16 00:54:49,929 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 00:54:49,945 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/.tmp/C/afa1a8d0db9c4fcfa46a1e30a3bbb4c3 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/C/afa1a8d0db9c4fcfa46a1e30a3bbb4c3 2023-04-16 00:54:49,950 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 5dcf6997c5a081edf1d95a6a373c6b08/C of 5dcf6997c5a081edf1d95a6a373c6b08 into afa1a8d0db9c4fcfa46a1e30a3bbb4c3(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 00:54:49,950 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5dcf6997c5a081edf1d95a6a373c6b08: 2023-04-16 00:54:49,950 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08., storeName=5dcf6997c5a081edf1d95a6a373c6b08/C, priority=11, startTime=1681606489475; duration=0sec 2023-04-16 00:54:49,950 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 00:54:50,003 DEBUG [Thread-1736] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:54:50,004 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:54:50,005 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:54:50,006 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:54:50,006 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:54:50,006 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:54:50,006 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:54:50,006 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:54:50,006 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:54:50,007 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:50,007 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:50,008 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:54:50,008 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:50,008 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:54:50,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:54:50,008 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:50,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:50,008 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:54:50,009 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:50,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:54:50,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:50,009 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:54:50,009 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:54:50,009 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:54:50,009 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:54:50,010 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:54:50,010 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:54:50,010 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:54:50,010 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:54:50,010 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. 2023-04-16 00:54:50,010 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. started... 2023-04-16 00:54:50,010 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. as already flushing 2023-04-16 00:54:50,010 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. 2023-04-16 00:54:50,010 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:54:50,010 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:54:50,010 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:50,010 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:54:50,010 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:54:50,013 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:50,013 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:50,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:50,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:50,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:50,013 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:50,013 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:54:50,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:50,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:50,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:50,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:50,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:50,031 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:54:50,031 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1848d6c7[Count = 0] remaining members to acquire global barrier 2023-04-16 00:54:50,031 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:54:50,031 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:50,032 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:50,032 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:50,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:50,032 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:54:50,032 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:54:50,032 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:54:50,032 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:50,032 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:54:50,034 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:50,034 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:54:50,034 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:54:50,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:50,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:50,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:50,034 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:54:50,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:50,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:50,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:50,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:50,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:50,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:50,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:50,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:54:50,036 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:54:50,036 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:54:50,036 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:54:50,036 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:54:50,036 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:54:50,036 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:54:50,038 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:50,038 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:50,038 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:50,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:50,038 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:54:50,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:50,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:50,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:50,038 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:54:50,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:54:50,038 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:50,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:50,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:50,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:50,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:50,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:50,040 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:50,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:50,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:50,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:50,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:50,043 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:54:50,043 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:54:50,043 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:50,043 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:54:50,043 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:54:50,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:54:50,043 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:54:50,043 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:50,043 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:54:50,043 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:54:50,043 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:50,043 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:50,043 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:50,043 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:50,044 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:50,044 DEBUG [Thread-1736] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:54:50,044 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:54:50,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:54:50,044 DEBUG [Thread-1736] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:54:50,124 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:50,124 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58306 deadline: 1681606550124, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:50,130 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:50,131 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58336 deadline: 1681606550130, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:50,132 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:50,133 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58282 deadline: 1681606550131, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:50,133 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:50,133 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58322 deadline: 1681606550131, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:50,144 DEBUG [Thread-1736] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:54:50,144 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:54:50,146 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:54:50,147 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:54:50,147 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:54:50,147 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:54:50,147 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:54:50,147 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:54:50,147 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:54:50,147 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:50,147 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:50,148 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:50,149 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:54:50,149 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:54:50,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:54:50,149 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:50,149 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:54:50,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:50,149 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:50,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:54:50,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:50,150 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:54:50,150 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:54:50,150 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:54:50,150 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:54:50,150 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:54:50,150 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:54:50,150 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:54:50,150 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:54:50,150 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. 2023-04-16 00:54:50,150 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. started... 2023-04-16 00:54:50,150 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. as already flushing 2023-04-16 00:54:50,150 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. 2023-04-16 00:54:50,150 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:54:50,150 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:54:50,150 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:50,150 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:54:50,150 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:54:50,152 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:50,152 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:50,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:50,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:50,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:50,153 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:50,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:50,153 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:54:50,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:50,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:50,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:50,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:50,154 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:54:50,154 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:54:50,154 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@67d0a7d4[Count = 0] remaining members to acquire global barrier 2023-04-16 00:54:50,154 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:50,155 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:50,155 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:50,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:50,155 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:50,155 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:54:50,155 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:54:50,155 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:54:50,155 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:54:50,158 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:54:50,158 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:54:50,158 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:50,158 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:54:50,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:50,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:50,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:50,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:50,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:50,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:50,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:50,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:50,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:50,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:50,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:54:50,160 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:54:50,160 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:54:50,161 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:54:50,161 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:54:50,161 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:54:50,161 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:54:50,162 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:50,162 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:50,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:50,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:50,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:50,162 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:50,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:50,162 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:54:50,162 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:50,162 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:54:50,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:50,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:54:50,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:50,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:50,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:50,164 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:50,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:50,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:50,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:50,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:50,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:50,166 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:50,166 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:50,166 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:50,166 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:54:50,166 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:54:50,166 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:54:50,166 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:54:50,166 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:54:50,166 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:50,166 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:50,166 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:50,166 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:50,166 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:54:50,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:54:50,166 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:54:50,166 DEBUG [Thread-1736] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:54:50,166 DEBUG [Thread-1736] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:54:50,167 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:54:50,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:54:50,197 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:50,197 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58292 deadline: 1681606550195, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:50,267 DEBUG [Thread-1736] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:54:50,267 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:54:50,269 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:54:50,270 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:54:50,270 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:54:50,270 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:54:50,270 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:54:50,270 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:54:50,270 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:54:50,270 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:50,270 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:50,271 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:54:50,271 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:50,271 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:54:50,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:54:50,272 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:50,272 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:54:50,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:50,272 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:50,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:54:50,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:50,273 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:54:50,273 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:54:50,273 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:54:50,273 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:54:50,273 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:54:50,273 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:54:50,273 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:54:50,273 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:54:50,273 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. 2023-04-16 00:54:50,273 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. started... 2023-04-16 00:54:50,273 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. as already flushing 2023-04-16 00:54:50,273 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. 2023-04-16 00:54:50,273 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:54:50,273 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:54:50,273 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:50,274 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:54:50,274 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:54:50,276 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:50,276 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:50,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:50,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:50,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:50,277 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:50,277 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:54:50,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:50,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:50,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:50,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:50,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:50,278 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:54:50,278 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@403ba269[Count = 0] remaining members to acquire global barrier 2023-04-16 00:54:50,278 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:54:50,278 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:50,279 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:50,279 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:50,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:50,280 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:54:50,280 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:54:50,280 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:54:50,280 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:50,280 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:54:50,281 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:50,281 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:54:50,281 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:54:50,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:50,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:50,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:50,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:50,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:50,281 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:54:50,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:50,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:50,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:50,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:50,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:50,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:54:50,284 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:54:50,284 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:54:50,284 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:54:50,284 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:54:50,284 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:54:50,284 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:54:50,286 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:50,286 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:50,286 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:50,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:50,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:50,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:50,286 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:54:50,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:50,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:50,286 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:50,286 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:54:50,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:54:50,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:50,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:50,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:50,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:50,287 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:50,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:50,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:50,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:50,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:50,293 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:50,293 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:54:50,293 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:50,293 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:54:50,293 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:54:50,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:54:50,293 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:54:50,293 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:54:50,293 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:50,294 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:50,294 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:50,294 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:54:50,294 DEBUG [Thread-1736] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:54:50,294 DEBUG [Thread-1736] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:54:50,293 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:54:50,294 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:54:50,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:54:50,294 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:50,294 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:50,334 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=100 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/.tmp/B/df50817c07ac46c1a3a80441a2bfdbf3 2023-04-16 00:54:50,355 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=100 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/.tmp/C/fbddbfa531154e4d8c66f90278d2c90a 2023-04-16 00:54:50,359 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/.tmp/A/ddfda75fb7554390b8b938dca019fdaa as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/A/ddfda75fb7554390b8b938dca019fdaa 2023-04-16 00:54:50,362 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/A/ddfda75fb7554390b8b938dca019fdaa, entries=200, sequenceid=100, filesize=13.9 K 2023-04-16 00:54:50,363 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/.tmp/B/df50817c07ac46c1a3a80441a2bfdbf3 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/B/df50817c07ac46c1a3a80441a2bfdbf3 2023-04-16 00:54:50,367 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/B/df50817c07ac46c1a3a80441a2bfdbf3, entries=150, sequenceid=100, filesize=11.6 K 2023-04-16 00:54:50,368 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/.tmp/C/fbddbfa531154e4d8c66f90278d2c90a as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/C/fbddbfa531154e4d8c66f90278d2c90a 2023-04-16 00:54:50,371 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/C/fbddbfa531154e4d8c66f90278d2c90a, entries=150, sequenceid=100, filesize=11.6 K 2023-04-16 00:54:50,372 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=67.09 KB/68700 for 5dcf6997c5a081edf1d95a6a373c6b08 in 877ms, sequenceid=100, compaction requested=false 2023-04-16 00:54:50,372 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 5dcf6997c5a081edf1d95a6a373c6b08: 2023-04-16 00:54:50,394 DEBUG [Thread-1736] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:54:50,394 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:54:50,396 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:54:50,397 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:54:50,397 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:54:50,397 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:54:50,397 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:54:50,397 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:54:50,397 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:54:50,397 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:50,397 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:50,398 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:50,398 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:54:50,399 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:54:50,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:54:50,399 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:50,399 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:54:50,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:50,399 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:50,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:54:50,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:50,399 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:54:50,400 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:54:50,400 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:54:50,400 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:54:50,400 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:54:50,400 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:54:50,400 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:54:50,400 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:54:50,400 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. 2023-04-16 00:54:50,400 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. started... 2023-04-16 00:54:50,400 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 5dcf6997c5a081edf1d95a6a373c6b08 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-16 00:54:50,400 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5dcf6997c5a081edf1d95a6a373c6b08, store=A 2023-04-16 00:54:50,400 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:54:50,401 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5dcf6997c5a081edf1d95a6a373c6b08, store=B 2023-04-16 00:54:50,401 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:54:50,401 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5dcf6997c5a081edf1d95a6a373c6b08, store=C 2023-04-16 00:54:50,401 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:54:50,631 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(9158): Flush requested on 5dcf6997c5a081edf1d95a6a373c6b08 2023-04-16 00:54:50,631 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. as already flushing 2023-04-16 00:54:50,672 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:50,673 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58306 deadline: 1681606550669, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:50,674 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:50,674 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58336 deadline: 1681606550671, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:50,676 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:50,676 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:50,676 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58322 deadline: 1681606550672, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:50,676 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58282 deadline: 1681606550673, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:50,774 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:50,775 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58306 deadline: 1681606550774, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:50,775 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:50,776 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58336 deadline: 1681606550775, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:50,778 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:50,778 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58322 deadline: 1681606550777, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:50,778 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:50,778 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58282 deadline: 1681606550777, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:50,814 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=116 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/.tmp/A/d4e6bc4b26f145a5a1616ea22cc689c2 2023-04-16 00:54:50,824 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=116 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/.tmp/B/ffec08e54d9c42539282ed2f59c3a937 2023-04-16 00:54:50,978 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:50,978 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58306 deadline: 1681606550976, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:50,979 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:50,979 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58336 deadline: 1681606550976, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:50,980 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:50,980 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58322 deadline: 1681606550979, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:50,982 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:50,982 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58282 deadline: 1681606550980, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:51,205 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:51,205 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58292 deadline: 1681606551204, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:51,233 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=116 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/.tmp/C/c11279046bc54aee9e0e44d59ebf98ac 2023-04-16 00:54:51,237 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/.tmp/A/d4e6bc4b26f145a5a1616ea22cc689c2 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/A/d4e6bc4b26f145a5a1616ea22cc689c2 2023-04-16 00:54:51,241 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/A/d4e6bc4b26f145a5a1616ea22cc689c2, entries=150, sequenceid=116, filesize=11.6 K 2023-04-16 00:54:51,241 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.StoreScanner(953): StoreScanner already has the close lock. There is no need to updateReaders 2023-04-16 00:54:51,242 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/.tmp/B/ffec08e54d9c42539282ed2f59c3a937 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/B/ffec08e54d9c42539282ed2f59c3a937 2023-04-16 00:54:51,245 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/B/ffec08e54d9c42539282ed2f59c3a937, entries=150, sequenceid=116, filesize=11.6 K 2023-04-16 00:54:51,246 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/.tmp/C/c11279046bc54aee9e0e44d59ebf98ac as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/C/c11279046bc54aee9e0e44d59ebf98ac 2023-04-16 00:54:51,250 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/C/c11279046bc54aee9e0e44d59ebf98ac, entries=150, sequenceid=116, filesize=11.6 K 2023-04-16 00:54:51,250 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=114.05 KB/116790 for 5dcf6997c5a081edf1d95a6a373c6b08 in 850ms, sequenceid=116, compaction requested=true 2023-04-16 00:54:51,251 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 5dcf6997c5a081edf1d95a6a373c6b08: 2023-04-16 00:54:51,251 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. 2023-04-16 00:54:51,251 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:54:51,251 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:54:51,251 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:51,254 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:54:51,254 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:54:51,256 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:51,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:51,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:51,256 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:51,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:51,256 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:51,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:51,256 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:54:51,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:51,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:51,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:51,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:51,257 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:54:51,257 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3707b0f1[Count = 0] remaining members to acquire global barrier 2023-04-16 00:54:51,258 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:54:51,258 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:51,258 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:51,259 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:51,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:51,259 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:54:51,259 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:51,259 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:54:51,259 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:54:51,259 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:54:51,261 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:54:51,261 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:51,261 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:54:51,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:51,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:51,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:51,261 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:54:51,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:51,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:51,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:51,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:51,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:51,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:51,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:51,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:54:51,263 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:54:51,263 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:54:51,263 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:54:51,263 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:54:51,264 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:54:51,264 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:54:51,265 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:51,265 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:51,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:51,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:51,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:51,265 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:51,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:51,265 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:54:51,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:51,265 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:54:51,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:54:51,266 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:51,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:51,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:51,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:51,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:51,267 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:51,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:51,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:51,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:51,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:51,269 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:54:51,269 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:51,269 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:54:51,269 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:54:51,269 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:54:51,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:54:51,269 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:54:51,269 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:54:51,269 DEBUG [Thread-1736] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:54:51,270 DEBUG [Thread-1736] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:54:51,269 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:54:51,269 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:51,270 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:51,270 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:51,270 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:51,270 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:51,270 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:51,271 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:54:51,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:54:51,281 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(9158): Flush requested on 5dcf6997c5a081edf1d95a6a373c6b08 2023-04-16 00:54:51,281 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 5dcf6997c5a081edf1d95a6a373c6b08 3/3 column families, dataSize=120.76 KB heapSize=360.05 KB 2023-04-16 00:54:51,281 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5dcf6997c5a081edf1d95a6a373c6b08, store=A 2023-04-16 00:54:51,282 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 00:54:51,282 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5dcf6997c5a081edf1d95a6a373c6b08, store=B 2023-04-16 00:54:51,282 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 00:54:51,282 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5dcf6997c5a081edf1d95a6a373c6b08, store=C 2023-04-16 00:54:51,282 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 00:54:51,292 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:51,292 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58306 deadline: 1681606551289, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:51,292 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:51,292 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58322 deadline: 1681606551290, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:51,293 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:51,293 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58282 deadline: 1681606551291, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:51,294 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:51,294 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58336 deadline: 1681606551291, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:51,370 DEBUG [Thread-1736] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:54:51,370 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:54:51,372 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:54:51,373 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:54:51,373 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:54:51,373 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:54:51,373 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:54:51,373 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:54:51,373 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:54:51,373 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:51,373 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:51,374 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:51,374 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:54:51,374 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:54:51,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:54:51,375 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:51,375 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:54:51,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:51,375 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:51,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:54:51,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:51,376 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:54:51,376 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:54:51,376 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:54:51,376 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:54:51,376 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:54:51,376 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:54:51,376 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:54:51,376 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:54:51,376 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. 2023-04-16 00:54:51,376 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. started... 2023-04-16 00:54:51,377 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. as already flushing 2023-04-16 00:54:51,377 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. 2023-04-16 00:54:51,377 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:54:51,377 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:54:51,377 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:51,377 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:54:51,377 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:54:51,379 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:51,379 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:51,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:51,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:51,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:51,379 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:51,379 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:54:51,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:51,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:51,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:51,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:51,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:51,380 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:54:51,381 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6927b638[Count = 0] remaining members to acquire global barrier 2023-04-16 00:54:51,381 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:54:51,381 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:51,382 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:51,382 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:51,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:51,382 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:54:51,382 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:54:51,382 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:54:51,382 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:51,382 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:54:51,383 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:54:51,383 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:51,383 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:54:51,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:51,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:51,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:51,383 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:54:51,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:51,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:51,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:51,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:51,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:51,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:51,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:51,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:54:51,386 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:54:51,386 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:54:51,386 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:54:51,386 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:54:51,386 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:54:51,386 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:54:51,387 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:51,387 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:51,387 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:51,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:51,387 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:54:51,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:51,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:51,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:51,388 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:54:51,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:54:51,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:51,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:51,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:51,388 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:51,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:51,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:51,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:51,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:51,389 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:51,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:51,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:51,392 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:51,392 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:54:51,392 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:51,392 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:54:51,392 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:51,392 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:54:51,392 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:54:51,392 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:54:51,392 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:54:51,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:54:51,392 DEBUG [Thread-1736] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:54:51,392 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:51,392 DEBUG [Thread-1736] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:54:51,392 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:51,393 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:51,393 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:51,393 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:54:51,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:54:51,395 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:51,395 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58322 deadline: 1681606551393, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:51,395 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:51,395 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58306 deadline: 1681606551393, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:51,395 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:51,395 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58282 deadline: 1681606551394, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:51,396 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:51,396 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58336 deadline: 1681606551395, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:51,493 DEBUG [Thread-1736] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:54:51,493 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:54:51,494 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:54:51,495 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:54:51,495 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:54:51,496 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:54:51,496 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:54:51,496 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:54:51,496 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:54:51,496 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:51,496 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:51,497 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:51,497 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:54:51,497 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:54:51,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:54:51,498 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:51,498 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:54:51,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:51,498 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:51,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:54:51,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:51,498 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:54:51,499 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:54:51,499 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:54:51,499 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:54:51,499 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:54:51,499 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:54:51,499 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:54:51,499 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:54:51,499 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. 2023-04-16 00:54:51,499 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. started... 2023-04-16 00:54:51,499 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. as already flushing 2023-04-16 00:54:51,499 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. 2023-04-16 00:54:51,499 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:54:51,499 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:54:51,499 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:51,499 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:54:51,499 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:54:51,501 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:51,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:51,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:51,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:51,501 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:51,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:51,501 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:51,501 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:54:51,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:51,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:51,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:51,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:51,502 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:54:51,502 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:54:51,502 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@bab52cf[Count = 0] remaining members to acquire global barrier 2023-04-16 00:54:51,502 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:51,504 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:51,504 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:51,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:51,504 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:54:51,504 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:54:51,504 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:54:51,504 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:51,504 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:54:51,506 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:54:51,506 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:51,506 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:54:51,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:51,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:51,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:51,506 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:54:51,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:51,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:51,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:51,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:51,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:51,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:51,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:51,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:54:51,508 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:54:51,508 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:54:51,509 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:54:51,509 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:54:51,509 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:54:51,509 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:54:51,510 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:51,510 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:51,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:51,510 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:51,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:51,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:51,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:51,510 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:54:51,511 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:54:51,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:51,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:54:51,511 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:51,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:51,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:51,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:51,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:51,512 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:51,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:51,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:51,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:51,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:51,516 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:54:51,516 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:54:51,516 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:51,516 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:54:51,516 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:54:51,516 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:54:51,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:54:51,516 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:51,516 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:54:51,516 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:51,517 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:51,517 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:51,517 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:51,517 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:51,517 DEBUG [Thread-1736] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:54:51,517 DEBUG [Thread-1736] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:54:51,517 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:54:51,517 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:54:51,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:54:51,598 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:51,598 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58322 deadline: 1681606551596, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:51,598 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:51,599 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58306 deadline: 1681606551597, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:51,599 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:51,599 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58282 deadline: 1681606551597, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:51,599 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:51,599 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58336 deadline: 1681606551597, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:51,617 DEBUG [Thread-1736] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:54:51,617 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:54:51,619 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:54:51,620 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:54:51,620 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:54:51,620 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:54:51,620 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:54:51,620 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:54:51,620 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:54:51,620 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:51,620 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:51,622 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:54:51,623 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:51,623 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:54:51,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:54:51,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:51,623 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:51,623 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:54:51,623 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:51,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:54:51,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:51,624 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:54:51,624 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:54:51,624 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:54:51,624 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:54:51,624 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:54:51,624 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:54:51,624 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:54:51,624 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:54:51,624 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. 2023-04-16 00:54:51,624 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. started... 2023-04-16 00:54:51,624 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. as already flushing 2023-04-16 00:54:51,624 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. 2023-04-16 00:54:51,625 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:54:51,625 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:54:51,625 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:51,625 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:54:51,625 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:54:51,626 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:51,626 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:51,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:51,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:51,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:51,626 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:51,626 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:54:51,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:51,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:51,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:51,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:51,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:51,628 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:54:51,628 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@161e9139[Count = 0] remaining members to acquire global barrier 2023-04-16 00:54:51,628 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:54:51,628 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:51,629 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:51,629 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:51,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:51,629 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:54:51,629 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:54:51,629 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:54:51,630 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:51,630 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:54:51,630 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:54:51,630 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:51,630 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:54:51,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:51,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:51,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:51,631 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:54:51,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:51,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:51,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:51,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:51,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:51,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:51,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:51,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:54:51,633 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:54:51,633 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:54:51,633 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:54:51,633 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:54:51,633 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:54:51,633 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:54:51,635 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:51,635 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:51,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:51,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:51,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:51,635 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:51,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:51,635 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:54:51,636 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:54:51,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:54:51,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:51,636 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:51,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:51,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:51,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:51,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:51,637 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:51,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:51,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:51,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:51,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:51,639 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:51,639 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:54:51,639 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:51,640 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:54:51,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:54:51,640 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:51,640 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:51,640 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:51,640 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:51,640 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:51,640 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:54:51,640 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:54:51,639 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:54:51,639 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:54:51,640 DEBUG [Thread-1736] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:54:51,640 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:54:51,640 DEBUG [Thread-1736] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:54:51,640 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:54:51,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:54:51,689 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=137 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/.tmp/A/14ae83c08fcc4fa58a3b06de5e07b03c 2023-04-16 00:54:51,740 DEBUG [Thread-1736] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:54:51,741 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:54:51,746 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:54:51,747 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:54:51,747 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:54:51,747 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:54:51,747 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:54:51,747 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:54:51,747 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:54:51,747 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:51,747 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:51,748 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:54:51,748 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:51,749 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:54:51,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:54:51,749 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:51,749 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:54:51,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:51,749 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:51,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:54:51,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:51,750 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:54:51,750 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:54:51,750 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:54:51,750 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:54:51,750 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:54:51,750 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:54:51,750 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:54:51,750 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:54:51,750 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. 2023-04-16 00:54:51,750 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. started... 2023-04-16 00:54:51,750 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. as already flushing 2023-04-16 00:54:51,750 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. 2023-04-16 00:54:51,750 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:54:51,751 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:54:51,751 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:51,751 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:54:51,751 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:54:51,752 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:51,752 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:51,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:51,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:51,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:51,752 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:51,752 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:54:51,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:51,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:51,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:51,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:51,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:51,753 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:54:51,754 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:54:51,754 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:51,754 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1c143b33[Count = 0] remaining members to acquire global barrier 2023-04-16 00:54:51,755 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:51,755 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:51,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:51,756 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:54:51,756 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:51,756 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:54:51,756 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:54:51,756 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:54:51,757 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:51,757 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:54:51,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:51,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:51,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:51,757 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:54:51,757 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:54:51,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:51,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:51,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:51,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:51,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:51,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:51,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:51,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:54:51,759 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:54:51,759 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:54:51,759 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:54:51,759 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:54:51,759 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:54:51,759 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:54:51,761 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:51,761 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:51,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:51,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:51,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:51,761 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:51,761 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:54:51,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:51,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:51,761 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:51,761 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:54:51,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:54:51,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:51,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:51,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:51,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:51,762 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:51,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:51,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:51,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:51,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:51,765 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:51,765 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:51,765 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:54:51,765 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:54:51,765 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:54:51,765 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:51,765 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:54:51,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:54:51,765 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:54:51,765 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:54:51,765 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:51,765 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:51,765 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:51,765 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:51,765 DEBUG [Thread-1736] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:54:51,765 DEBUG [Thread-1736] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:54:51,765 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:54:51,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:54:51,865 DEBUG [Thread-1736] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:54:51,866 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:54:51,867 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:54:51,868 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:54:51,868 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:54:51,868 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:54:51,868 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:54:51,868 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:54:51,868 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:54:51,869 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:51,869 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:51,871 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:54:51,871 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:51,871 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:54:51,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:54:51,871 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:51,871 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:54:51,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:51,872 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:51,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:54:51,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:51,872 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:54:51,872 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:54:51,872 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:54:51,872 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:54:51,872 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:54:51,872 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:54:51,873 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:54:51,873 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:54:51,873 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. 2023-04-16 00:54:51,873 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. started... 2023-04-16 00:54:51,873 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. as already flushing 2023-04-16 00:54:51,873 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. 2023-04-16 00:54:51,873 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:54:51,873 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:54:51,873 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:51,873 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:54:51,873 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:54:51,875 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:51,875 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:51,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:51,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:51,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:51,875 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:51,875 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:54:51,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:51,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:51,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:51,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:51,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:51,877 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:54:51,877 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@48d6d8ff[Count = 0] remaining members to acquire global barrier 2023-04-16 00:54:51,877 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:54:51,877 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:51,878 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:51,878 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:51,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:51,878 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:54:51,878 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:51,878 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:54:51,878 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:54:51,878 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:54:51,879 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:54:51,879 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:51,879 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:54:51,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:51,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:51,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:51,880 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:54:51,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:51,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:51,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:51,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:51,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:51,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:51,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:51,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:54:51,882 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:54:51,882 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:54:51,882 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:54:51,882 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:54:51,882 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:54:51,882 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:54:51,885 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:51,885 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:51,885 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:51,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:51,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:51,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:51,885 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:54:51,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:51,885 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:51,885 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:54:51,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:54:51,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:51,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:51,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:51,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:51,886 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:51,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:51,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:51,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:51,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:51,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:51,888 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:51,889 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:54:51,889 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:51,889 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:54:51,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:54:51,889 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:54:51,889 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:54:51,889 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:51,889 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:54:51,889 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:54:51,889 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:54:51,889 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:54:51,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:54:51,889 DEBUG [Thread-1736] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:54:51,889 DEBUG [Thread-1736] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:54:51,889 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:51,889 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:51,889 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:51,889 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:51,901 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:51,901 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58322 deadline: 1681606551899, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:51,901 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:51,901 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58306 deadline: 1681606551899, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:51,902 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:51,902 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58282 deadline: 1681606551900, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:51,902 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:51,902 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58336 deadline: 1681606551901, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:51,989 DEBUG [Thread-1736] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:54:51,990 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:54:51,991 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:54:51,992 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:54:51,992 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:54:51,992 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:54:51,992 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:54:51,992 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:54:51,992 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:54:51,993 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:51,993 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:52,000 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:54:52,000 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:52,001 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:54:52,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:54:52,001 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:52,001 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:54:52,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:52,001 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:52,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:54:52,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:52,002 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:54:52,002 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:54:52,002 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:54:52,002 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:54:52,002 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:54:52,002 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:54:52,002 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:54:52,002 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:54:52,002 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. 2023-04-16 00:54:52,002 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. started... 2023-04-16 00:54:52,002 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. as already flushing 2023-04-16 00:54:52,002 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. 2023-04-16 00:54:52,003 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:54:52,003 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:54:52,003 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:52,003 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:54:52,003 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:54:52,005 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:52,005 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:52,005 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:52,005 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:54:52,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:52,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:52,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:52,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:52,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:52,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:52,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:52,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:52,008 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:54:52,008 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@19bac600[Count = 0] remaining members to acquire global barrier 2023-04-16 00:54:52,008 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:54:52,009 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:52,010 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:52,010 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:52,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:52,010 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:54:52,010 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:54:52,010 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:52,010 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:54:52,010 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:54:52,011 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:54:52,011 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:52,011 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:54:52,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:52,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:52,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:52,011 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:54:52,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:52,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:52,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:52,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:52,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:52,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:52,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:52,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:54:52,014 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:54:52,014 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:54:52,014 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:54:52,014 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:54:52,014 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:54:52,014 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:54:52,015 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:52,015 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:52,015 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:52,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:52,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:52,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:52,015 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:54:52,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:52,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:52,016 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:54:52,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:54:52,016 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:52,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:52,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:52,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:52,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:52,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:52,017 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:52,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:52,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:52,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:52,020 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:52,020 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:54:52,020 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:52,020 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:54:52,020 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:54:52,021 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:54:52,020 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:54:52,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:54:52,020 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:54:52,020 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:52,021 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:52,021 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:52,021 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:52,021 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:52,021 DEBUG [Thread-1736] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:54:52,021 DEBUG [Thread-1736] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:54:52,021 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:54:52,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:54:52,100 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=137 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/.tmp/B/fa08ea119d3b4da48d98ea0fa4dbf149 2023-04-16 00:54:52,121 DEBUG [Thread-1736] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:54:52,122 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:54:52,123 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:54:52,124 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:54:52,124 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:54:52,124 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:54:52,124 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:54:52,124 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:54:52,124 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:54:52,125 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:52,125 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:52,126 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:52,126 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:54:52,126 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:54:52,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:54:52,126 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:52,127 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:54:52,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:52,127 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:52,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:54:52,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:52,127 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:54:52,127 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:54:52,128 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:54:52,128 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:54:52,128 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:54:52,128 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:54:52,128 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:54:52,128 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:54:52,128 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. 2023-04-16 00:54:52,128 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. started... 2023-04-16 00:54:52,128 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. as already flushing 2023-04-16 00:54:52,128 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. 2023-04-16 00:54:52,128 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:54:52,128 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:54:52,128 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:52,128 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:54:52,128 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:54:52,130 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:52,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:52,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:52,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:52,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:52,130 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:52,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:52,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:52,130 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:52,130 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:54:52,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:52,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:52,131 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:54:52,131 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7b4f0b57[Count = 0] remaining members to acquire global barrier 2023-04-16 00:54:52,131 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:54:52,131 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:52,132 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:52,133 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:52,133 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:54:52,133 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:52,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:52,133 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:54:52,133 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:54:52,133 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:54:52,135 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:52,135 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:54:52,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:52,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:52,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:52,135 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:54:52,136 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:54:52,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:52,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:52,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:52,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:52,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:52,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:52,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:52,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:54:52,138 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:54:52,138 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:54:52,138 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:54:52,138 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:54:52,138 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:54:52,138 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:54:52,139 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:52,139 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:52,139 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:52,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:52,139 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:54:52,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:52,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:52,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:52,140 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:54:52,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:54:52,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:52,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:52,140 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:52,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:52,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:52,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:52,141 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:52,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:52,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:52,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:52,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:52,143 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:52,143 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:54:52,143 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:52,143 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:54:52,143 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:52,143 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:54:52,143 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:54:52,143 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:54:52,143 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:54:52,143 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:52,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:54:52,143 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:54:52,144 DEBUG [Thread-1736] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:54:52,144 DEBUG [Thread-1736] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:54:52,144 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:52,144 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:52,144 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:52,144 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:54:52,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:54:52,244 DEBUG [Thread-1736] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:54:52,244 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:54:52,246 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:54:52,247 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:54:52,247 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:54:52,247 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:54:52,247 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:54:52,247 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:54:52,247 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:54:52,248 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:52,248 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:52,249 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:54:52,249 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:52,249 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:54:52,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:54:52,249 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:52,249 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:54:52,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:52,250 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:52,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:54:52,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:52,250 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:54:52,250 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:54:52,250 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:54:52,250 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:54:52,250 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:54:52,250 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:54:52,251 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:54:52,251 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:54:52,251 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. 2023-04-16 00:54:52,251 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. started... 2023-04-16 00:54:52,251 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. as already flushing 2023-04-16 00:54:52,251 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. 2023-04-16 00:54:52,251 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:54:52,251 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:54:52,251 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:52,251 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:54:52,251 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:54:52,253 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:52,253 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:52,253 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:52,254 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:54:52,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:52,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:52,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:52,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:52,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:52,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:52,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:52,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:52,255 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:54:52,255 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@463f6aae[Count = 0] remaining members to acquire global barrier 2023-04-16 00:54:52,255 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:54:52,255 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:52,256 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:52,256 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:52,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:52,256 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:54:52,256 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:54:52,256 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:54:52,256 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:52,256 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:54:52,257 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:52,257 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:54:52,258 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:54:52,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:52,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:52,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:52,258 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:54:52,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:52,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:52,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:52,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:52,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:52,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:52,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:52,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:54:52,260 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:54:52,260 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:54:52,260 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:54:52,260 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:54:52,260 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:54:52,260 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:54:52,261 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:52,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:52,261 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:52,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:52,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:52,262 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:52,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:52,262 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:54:52,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:52,262 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:54:52,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:54:52,262 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:52,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:52,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:52,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:52,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:52,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:52,263 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:52,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:52,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:52,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:52,266 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:52,267 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:52,267 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:52,267 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:52,267 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:52,267 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:52,267 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:52,267 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:54:52,267 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:54:52,267 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:54:52,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:54:52,267 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:54:52,267 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:54:52,267 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:54:52,267 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:54:52,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:54:52,267 DEBUG [Thread-1736] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:54:52,267 DEBUG [Thread-1736] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:54:52,367 DEBUG [Thread-1736] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:54:52,368 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:54:52,369 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:54:52,370 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:54:52,371 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:54:52,371 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:54:52,371 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:54:52,371 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:54:52,371 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:54:52,371 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:52,371 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:52,373 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:52,373 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:54:52,373 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:54:52,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:54:52,373 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:52,373 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:54:52,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:52,373 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:52,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:54:52,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:52,374 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:54:52,374 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:54:52,374 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:54:52,374 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:54:52,374 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:54:52,374 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:54:52,374 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:54:52,374 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:54:52,374 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. 2023-04-16 00:54:52,375 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. started... 2023-04-16 00:54:52,375 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. as already flushing 2023-04-16 00:54:52,375 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. 2023-04-16 00:54:52,375 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:54:52,375 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:54:52,375 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:52,375 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:54:52,375 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:54:52,382 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:52,382 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:52,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:52,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:52,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:52,382 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:52,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:52,382 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:54:52,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:52,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:52,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:52,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:52,383 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:54:52,383 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@38d043b6[Count = 0] remaining members to acquire global barrier 2023-04-16 00:54:52,383 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:54:52,384 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:52,385 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:52,385 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:52,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:52,385 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:52,386 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:54:52,386 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:54:52,386 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:54:52,386 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:54:52,387 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:54:52,387 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:54:52,387 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:52,387 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:54:52,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:52,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:52,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:52,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:52,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:52,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:52,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:52,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:52,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:52,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:52,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:54:52,390 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:54:52,390 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:54:52,390 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:54:52,390 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:54:52,390 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:54:52,390 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:54:52,391 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:52,391 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:54:52,391 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:52,391 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:52,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:52,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:52,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:52,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:52,392 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:54:52,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:54:52,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:52,392 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:52,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:52,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:52,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:52,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:52,393 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:52,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:52,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:52,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:52,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:52,395 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:52,395 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:52,395 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:52,395 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:52,395 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:52,395 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:54:52,395 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:54:52,395 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:54:52,395 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:54:52,395 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:54:52,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:54:52,395 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:54:52,395 DEBUG [Thread-1736] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:54:52,395 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:52,395 DEBUG [Thread-1736] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:54:52,395 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:54:52,396 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:54:52,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:54:52,396 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:52,404 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:52,404 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58306 deadline: 1681606552403, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:52,404 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:52,404 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58282 deadline: 1681606552403, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:52,406 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:52,406 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58336 deadline: 1681606552404, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:52,407 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:52,407 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58322 deadline: 1681606552406, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:52,496 DEBUG [Thread-1736] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:54:52,496 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:54:52,498 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:54:52,499 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:54:52,499 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:54:52,499 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:54:52,499 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:54:52,499 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:54:52,499 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:54:52,499 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:52,499 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:52,501 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:54:52,501 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:52,501 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:54:52,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:54:52,502 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:52,502 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:54:52,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:52,502 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:52,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:54:52,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:52,503 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:54:52,503 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:54:52,503 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:54:52,503 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:54:52,503 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:54:52,503 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:54:52,503 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:54:52,503 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:54:52,503 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. 2023-04-16 00:54:52,503 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. started... 2023-04-16 00:54:52,503 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. as already flushing 2023-04-16 00:54:52,503 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. 2023-04-16 00:54:52,503 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:54:52,503 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:54:52,504 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:52,504 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:54:52,504 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:54:52,505 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:52,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:52,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:52,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:52,505 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:52,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:52,505 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:52,505 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:54:52,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:52,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:52,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:52,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:52,507 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:54:52,507 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7795086a[Count = 0] remaining members to acquire global barrier 2023-04-16 00:54:52,507 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:54:52,507 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:52,508 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:52,508 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:52,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:52,508 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:54:52,508 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:54:52,508 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:54:52,508 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:52,508 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:54:52,509 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:54:52,510 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:54:52,509 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:52,510 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:54:52,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:52,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:52,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:52,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:52,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:52,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:52,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:52,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:52,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:52,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:52,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:54:52,512 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:54:52,512 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:54:52,512 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:54:52,512 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:54:52,512 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:54:52,512 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:54:52,515 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:52,515 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:52,515 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:52,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:52,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:52,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:52,515 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:54:52,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:52,516 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:52,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:52,516 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:54:52,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:54:52,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:52,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:52,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:52,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:52,517 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:52,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:52,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:52,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:52,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:52,518 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=137 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/.tmp/C/ad87da1fbdfa4b3e9215a86ce90d710a 2023-04-16 00:54:52,521 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:52,521 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:52,521 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:52,521 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:54:52,521 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:52,521 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:54:52,521 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:54:52,521 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:54:52,521 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:54:52,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:54:52,521 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:54:52,521 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:54:52,521 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:52,521 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:52,521 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:52,521 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:54:52,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:54:52,521 DEBUG [Thread-1736] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:54:52,522 DEBUG [Thread-1736] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:54:52,525 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/.tmp/A/14ae83c08fcc4fa58a3b06de5e07b03c as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/A/14ae83c08fcc4fa58a3b06de5e07b03c 2023-04-16 00:54:52,528 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/A/14ae83c08fcc4fa58a3b06de5e07b03c, entries=150, sequenceid=137, filesize=11.8 K 2023-04-16 00:54:52,529 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/.tmp/B/fa08ea119d3b4da48d98ea0fa4dbf149 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/B/fa08ea119d3b4da48d98ea0fa4dbf149 2023-04-16 00:54:52,532 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/B/fa08ea119d3b4da48d98ea0fa4dbf149, entries=150, sequenceid=137, filesize=11.8 K 2023-04-16 00:54:52,533 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/.tmp/C/ad87da1fbdfa4b3e9215a86ce90d710a as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/C/ad87da1fbdfa4b3e9215a86ce90d710a 2023-04-16 00:54:52,537 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/C/ad87da1fbdfa4b3e9215a86ce90d710a, entries=150, sequenceid=137, filesize=11.8 K 2023-04-16 00:54:52,537 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~120.76 KB/123660, heapSize ~360 KB/368640, currentSize=53.67 KB/54960 for 5dcf6997c5a081edf1d95a6a373c6b08 in 1256ms, sequenceid=137, compaction requested=true 2023-04-16 00:54:52,537 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 5dcf6997c5a081edf1d95a6a373c6b08: 2023-04-16 00:54:52,537 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 00:54:52,538 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-16 00:54:52,538 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-16 00:54:52,538 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 00:54:52,538 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 00:54:52,539 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 50206 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-16 00:54:52,539 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HStore(1912): 5dcf6997c5a081edf1d95a6a373c6b08/A is initiating minor compaction (all files) 2023-04-16 00:54:52,539 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 5dcf6997c5a081edf1d95a6a373c6b08/A in TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. 2023-04-16 00:54:52,539 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/A/d100f6235c09495fa58b0f7fa41a0a28, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/A/ddfda75fb7554390b8b938dca019fdaa, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/A/d4e6bc4b26f145a5a1616ea22cc689c2, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/A/14ae83c08fcc4fa58a3b06de5e07b03c] into tmpdir=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/.tmp, totalSize=49.0 K 2023-04-16 00:54:52,539 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47866 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-16 00:54:52,539 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HStore(1912): 5dcf6997c5a081edf1d95a6a373c6b08/B is initiating minor compaction (all files) 2023-04-16 00:54:52,539 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 5dcf6997c5a081edf1d95a6a373c6b08/B in TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. 2023-04-16 00:54:52,539 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/B/1699ebb87e4844ca9710c20f85a137ee, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/B/df50817c07ac46c1a3a80441a2bfdbf3, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/B/ffec08e54d9c42539282ed2f59c3a937, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/B/fa08ea119d3b4da48d98ea0fa4dbf149] into tmpdir=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/.tmp, totalSize=46.7 K 2023-04-16 00:54:52,540 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting 1699ebb87e4844ca9710c20f85a137ee, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=80, earliestPutTs=1681606488396 2023-04-16 00:54:52,540 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.Compactor(207): Compacting d100f6235c09495fa58b0f7fa41a0a28, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=80, earliestPutTs=1681606488396 2023-04-16 00:54:52,541 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting df50817c07ac46c1a3a80441a2bfdbf3, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=100, earliestPutTs=1681606489064 2023-04-16 00:54:52,541 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.Compactor(207): Compacting ddfda75fb7554390b8b938dca019fdaa, keycount=200, bloomtype=ROW, size=13.9 K, encoding=NONE, compression=NONE, seqNum=100, earliestPutTs=1681606489064 2023-04-16 00:54:52,541 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting ffec08e54d9c42539282ed2f59c3a937, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=116, earliestPutTs=1681606489502 2023-04-16 00:54:52,541 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.Compactor(207): Compacting d4e6bc4b26f145a5a1616ea22cc689c2, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=116, earliestPutTs=1681606489502 2023-04-16 00:54:52,541 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting fa08ea119d3b4da48d98ea0fa4dbf149, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=137, earliestPutTs=1681606490643 2023-04-16 00:54:52,542 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.Compactor(207): Compacting 14ae83c08fcc4fa58a3b06de5e07b03c, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=137, earliestPutTs=1681606490643 2023-04-16 00:54:52,550 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] throttle.PressureAwareThroughputController(145): 5dcf6997c5a081edf1d95a6a373c6b08#A#compaction#243 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-16 00:54:52,552 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] throttle.PressureAwareThroughputController(145): 5dcf6997c5a081edf1d95a6a373c6b08#B#compaction#244 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-16 00:54:52,622 DEBUG [Thread-1736] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:54:52,622 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:54:52,624 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:54:52,625 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:54:52,625 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:54:52,625 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:54:52,625 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:54:52,625 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:54:52,625 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:54:52,625 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:52,626 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:52,627 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:52,627 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:54:52,627 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:54:52,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:54:52,627 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:52,627 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:54:52,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:52,627 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:52,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:54:52,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:52,628 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:54:52,628 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:54:52,628 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:54:52,628 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:54:52,628 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:54:52,628 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:54:52,628 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:54:52,628 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:54:52,628 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. 2023-04-16 00:54:52,629 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. started... 2023-04-16 00:54:52,629 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 5dcf6997c5a081edf1d95a6a373c6b08 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-16 00:54:52,629 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5dcf6997c5a081edf1d95a6a373c6b08, store=A 2023-04-16 00:54:52,629 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:54:52,629 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5dcf6997c5a081edf1d95a6a373c6b08, store=B 2023-04-16 00:54:52,629 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:54:52,629 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5dcf6997c5a081edf1d95a6a373c6b08, store=C 2023-04-16 00:54:52,629 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:54:52,971 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/.tmp/A/96b5fc95057744639455f4bc8f1fa666 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/A/96b5fc95057744639455f4bc8f1fa666 2023-04-16 00:54:52,972 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/.tmp/B/720dc9d7aa6645c4a9c332a498a4e86d as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/B/720dc9d7aa6645c4a9c332a498a4e86d 2023-04-16 00:54:52,976 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 5dcf6997c5a081edf1d95a6a373c6b08/B of 5dcf6997c5a081edf1d95a6a373c6b08 into 720dc9d7aa6645c4a9c332a498a4e86d(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 00:54:52,976 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5dcf6997c5a081edf1d95a6a373c6b08: 2023-04-16 00:54:52,976 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08., storeName=5dcf6997c5a081edf1d95a6a373c6b08/B, priority=12, startTime=1681606492538; duration=0sec 2023-04-16 00:54:52,976 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 5dcf6997c5a081edf1d95a6a373c6b08/A of 5dcf6997c5a081edf1d95a6a373c6b08 into 96b5fc95057744639455f4bc8f1fa666(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 00:54:52,976 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 00:54:52,976 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5dcf6997c5a081edf1d95a6a373c6b08: 2023-04-16 00:54:52,976 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08., storeName=5dcf6997c5a081edf1d95a6a373c6b08/A, priority=12, startTime=1681606492537; duration=0sec 2023-04-16 00:54:52,976 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-16 00:54:52,976 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 00:54:52,977 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47866 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-16 00:54:52,977 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HStore(1912): 5dcf6997c5a081edf1d95a6a373c6b08/C is initiating minor compaction (all files) 2023-04-16 00:54:52,977 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 5dcf6997c5a081edf1d95a6a373c6b08/C in TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. 2023-04-16 00:54:52,977 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/C/afa1a8d0db9c4fcfa46a1e30a3bbb4c3, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/C/fbddbfa531154e4d8c66f90278d2c90a, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/C/c11279046bc54aee9e0e44d59ebf98ac, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/C/ad87da1fbdfa4b3e9215a86ce90d710a] into tmpdir=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/.tmp, totalSize=46.7 K 2023-04-16 00:54:52,978 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting afa1a8d0db9c4fcfa46a1e30a3bbb4c3, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=80, earliestPutTs=1681606488396 2023-04-16 00:54:52,978 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting fbddbfa531154e4d8c66f90278d2c90a, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=100, earliestPutTs=1681606489064 2023-04-16 00:54:52,978 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting c11279046bc54aee9e0e44d59ebf98ac, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=116, earliestPutTs=1681606489502 2023-04-16 00:54:52,979 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting ad87da1fbdfa4b3e9215a86ce90d710a, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=137, earliestPutTs=1681606490643 2023-04-16 00:54:52,994 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] throttle.PressureAwareThroughputController(145): 5dcf6997c5a081edf1d95a6a373c6b08#C#compaction#246 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-16 00:54:53,013 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/.tmp/C/96e1a998284c456aa0504ba6f47358e3 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/C/96e1a998284c456aa0504ba6f47358e3 2023-04-16 00:54:53,018 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 5dcf6997c5a081edf1d95a6a373c6b08/C of 5dcf6997c5a081edf1d95a6a373c6b08 into 96e1a998284c456aa0504ba6f47358e3(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 00:54:53,018 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5dcf6997c5a081edf1d95a6a373c6b08: 2023-04-16 00:54:53,018 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08., storeName=5dcf6997c5a081edf1d95a6a373c6b08/C, priority=12, startTime=1681606492538; duration=0sec 2023-04-16 00:54:53,018 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 00:54:53,037 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=148 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/.tmp/A/3ce09f4850a94551ab78a00899f062b9 2023-04-16 00:54:53,046 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=148 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/.tmp/B/c4ca91c939e447aa983f1eebb0c4eccf 2023-04-16 00:54:53,057 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=148 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/.tmp/C/b60e05604a59435fb573ac3f216f8a22 2023-04-16 00:54:53,061 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/.tmp/A/3ce09f4850a94551ab78a00899f062b9 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/A/3ce09f4850a94551ab78a00899f062b9 2023-04-16 00:54:53,065 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/A/3ce09f4850a94551ab78a00899f062b9, entries=150, sequenceid=148, filesize=11.8 K 2023-04-16 00:54:53,066 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/.tmp/B/c4ca91c939e447aa983f1eebb0c4eccf as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/B/c4ca91c939e447aa983f1eebb0c4eccf 2023-04-16 00:54:53,074 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/B/c4ca91c939e447aa983f1eebb0c4eccf, entries=150, sequenceid=148, filesize=11.8 K 2023-04-16 00:54:53,074 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/.tmp/C/b60e05604a59435fb573ac3f216f8a22 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/C/b60e05604a59435fb573ac3f216f8a22 2023-04-16 00:54:53,080 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/C/b60e05604a59435fb573ac3f216f8a22, entries=150, sequenceid=148, filesize=11.8 K 2023-04-16 00:54:53,080 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.StoreScanner(961): StoreScanner already closing. There is no need to updateReaders 2023-04-16 00:54:53,081 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=0 B/0 for 5dcf6997c5a081edf1d95a6a373c6b08 in 452ms, sequenceid=148, compaction requested=false 2023-04-16 00:54:53,081 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 5dcf6997c5a081edf1d95a6a373c6b08: 2023-04-16 00:54:53,081 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. 2023-04-16 00:54:53,081 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:54:53,081 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:54:53,081 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:53,081 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:54:53,081 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:54:53,083 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:53,083 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:53,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:53,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:53,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:53,083 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:53,083 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:54:53,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:53,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:53,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:53,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:53,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:53,085 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:54:53,085 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7483b07c[Count = 0] remaining members to acquire global barrier 2023-04-16 00:54:53,085 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:54:53,085 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:53,087 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:53,087 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:53,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:53,088 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:54:53,088 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:54:53,088 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:54:53,088 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:53,088 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:54:53,089 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:54:53,089 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:53,089 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:54:53,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:53,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:53,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:53,089 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:54:53,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:53,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:53,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:53,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:53,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:53,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:53,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:53,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:54:53,092 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:54:53,092 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:54:53,092 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:54:53,092 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:54:53,092 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:54:53,092 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:54:53,093 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:53,093 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:53,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:53,094 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:53,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:53,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:53,094 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:54:53,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:53,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:53,094 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:54:53,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:54:53,094 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:53,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:53,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:53,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:53,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:53,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:53,096 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:53,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:53,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:53,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:53,098 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:54:53,098 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:53,098 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:54:53,098 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:54:53,099 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:54:53,099 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:54:53,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:54:53,099 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:54:53,099 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:54:53,098 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:53,099 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:53,099 DEBUG [Thread-1736] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:54:53,099 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:53,099 DEBUG [Thread-1736] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:54:53,099 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:53,099 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:53,099 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:53,099 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:54:53,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:54:53,199 DEBUG [Thread-1736] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:54:53,200 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:54:53,201 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:54:53,202 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:54:53,202 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:54:53,203 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:54:53,203 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:54:53,203 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:54:53,203 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:54:53,203 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:53,203 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:53,205 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:53,205 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:54:53,205 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:54:53,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:54:53,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:53,205 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:53,205 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:54:53,206 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:53,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:54:53,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:53,206 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:54:53,206 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:54:53,206 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:54:53,206 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:54:53,206 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:54:53,206 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:54:53,207 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:54:53,207 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:54:53,207 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. 2023-04-16 00:54:53,207 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. started... 2023-04-16 00:54:53,207 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 5dcf6997c5a081edf1d95a6a373c6b08: 2023-04-16 00:54:53,207 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. 2023-04-16 00:54:53,207 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:54:53,207 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:54:53,207 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:53,207 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:54:53,207 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:54:53,209 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:53,209 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:53,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:53,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:53,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:53,209 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:53,209 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:54:53,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:53,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:53,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:53,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:53,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:53,210 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:54:53,211 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@49d4de58[Count = 0] remaining members to acquire global barrier 2023-04-16 00:54:53,211 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:54:53,211 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:53,212 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:53,212 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:53,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:53,212 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:54:53,212 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:53,212 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:54:53,212 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:54:53,212 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:54:53,217 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:53,217 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:54:53,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:53,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:53,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:53,217 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:54:53,218 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:54:53,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:53,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:53,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:53,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:53,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:53,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:53,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:53,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:54:53,220 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:54:53,220 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:54:53,220 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:54:53,220 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:54:53,220 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:54:53,220 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:54:53,222 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:53,222 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:53,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:53,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:53,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:53,222 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:53,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:53,222 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:54:53,222 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:54:53,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:54:53,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:53,223 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:53,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:53,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:53,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:53,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:53,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:53,224 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:53,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:53,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:53,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:53,226 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:53,226 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:53,226 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:53,226 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:54:53,226 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:54:53,226 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:54:53,226 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:54:53,226 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:53,227 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:53,227 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:53,227 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:53,226 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:54:53,227 DEBUG [Thread-1736] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:54:53,226 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:54:53,226 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:54:53,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:54:53,227 DEBUG [Thread-1736] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:54:53,227 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:54:53,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:54:53,244 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(9158): Flush requested on 5dcf6997c5a081edf1d95a6a373c6b08 2023-04-16 00:54:53,244 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 5dcf6997c5a081edf1d95a6a373c6b08 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-16 00:54:53,244 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5dcf6997c5a081edf1d95a6a373c6b08, store=A 2023-04-16 00:54:53,244 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:54:53,244 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5dcf6997c5a081edf1d95a6a373c6b08, store=B 2023-04-16 00:54:53,244 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:54:53,244 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5dcf6997c5a081edf1d95a6a373c6b08, store=C 2023-04-16 00:54:53,244 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:54:53,268 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=162 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/.tmp/A/395a461575f54884afaddaeac4a27715 2023-04-16 00:54:53,294 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681606493281, max timestamp=1681606493292Num uniques 859; 2023-04-16 00:54:53,294 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-16 00:54:53,294 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681606493281, max timestamp=1681606493292Num uniques 859; 2023-04-16 00:54:53,294 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-16 00:54:53,294 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681606493281, max timestamp=1681606493292Num uniques 859; 2023-04-16 00:54:53,294 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-16 00:54:53,327 DEBUG [Thread-1736] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:54:53,328 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:54:53,329 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:54:53,330 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:54:53,330 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:54:53,330 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:54:53,330 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:54:53,330 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:54:53,330 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:54:53,331 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:53,331 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:53,332 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:54:53,332 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:53,332 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:54:53,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:54:53,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:53,333 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:53,333 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:54:53,333 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:53,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:54:53,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:53,333 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:54:53,333 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:54:53,333 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:54:53,333 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:54:53,334 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:54:53,334 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:54:53,334 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:54:53,334 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:54:53,334 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. 2023-04-16 00:54:53,334 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. started... 2023-04-16 00:54:53,334 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. as already flushing 2023-04-16 00:54:53,334 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. 2023-04-16 00:54:53,334 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:54:53,334 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:54:53,334 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:53,334 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:54:53,334 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:54:53,337 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:53,337 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:53,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:53,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:53,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:53,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:53,337 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:53,337 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:54:53,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:53,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:53,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:53,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:53,339 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:54:53,339 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@73a3f927[Count = 0] remaining members to acquire global barrier 2023-04-16 00:54:53,339 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:54:53,339 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:53,340 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:53,340 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:53,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:53,340 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:54:53,340 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:54:53,340 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:54:53,340 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:53,341 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:54:53,342 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:54:53,342 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:54:53,342 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:53,342 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:54:53,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:53,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:53,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:53,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:53,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:53,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:53,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:53,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:53,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:53,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:53,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:54:53,345 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:54:53,345 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:54:53,345 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:54:53,345 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:54:53,345 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:54:53,345 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:54:53,346 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:53,346 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:53,346 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:53,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:53,346 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:54:53,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:53,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:53,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:53,347 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:53,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:53,347 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:54:53,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:54:53,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:53,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:53,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:53,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:53,348 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:53,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:53,348 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:53,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:53,348 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58292 deadline: 1681606553346, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:53,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:53,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:53,355 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:54:53,355 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:53,355 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:54:53,355 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:54:53,355 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:54:53,355 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:54:53,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:54:53,356 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:54:53,356 DEBUG [Thread-1736] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:54:53,355 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:54:53,355 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:53,356 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:53,356 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:53,356 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:53,356 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:53,356 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:53,356 DEBUG [Thread-1736] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:54:53,356 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:54:53,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:54:53,413 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:53,413 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58282 deadline: 1681606553411, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:53,415 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:53,415 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58306 deadline: 1681606553412, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:53,417 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:53,417 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58336 deadline: 1681606553415, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:53,417 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:53,417 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58322 deadline: 1681606553416, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:53,451 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:53,451 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58292 deadline: 1681606553449, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:53,456 DEBUG [Thread-1736] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:54:53,457 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:54:53,458 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:54:53,459 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:54:53,459 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:54:53,459 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:54:53,459 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:54:53,459 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:54:53,459 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:54:53,460 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:53,460 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:53,461 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:54:53,461 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:53,461 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:54:53,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:54:53,461 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:53,462 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:54:53,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:53,462 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:53,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:54:53,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:53,462 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:54:53,462 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:54:53,463 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:54:53,463 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:54:53,463 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:54:53,463 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:54:53,463 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:54:53,463 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:54:53,463 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. 2023-04-16 00:54:53,463 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. started... 2023-04-16 00:54:53,463 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. as already flushing 2023-04-16 00:54:53,463 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. 2023-04-16 00:54:53,463 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:54:53,463 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:54:53,463 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:53,463 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:54:53,463 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:54:53,465 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:53,465 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:53,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:53,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:53,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:53,465 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:53,465 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:54:53,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:53,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:53,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:53,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:53,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:53,466 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:54:53,466 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:54:53,466 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2c2a7961[Count = 0] remaining members to acquire global barrier 2023-04-16 00:54:53,466 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:53,467 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:53,467 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:53,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:53,468 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:54:53,468 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:54:53,468 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:53,468 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:54:53,468 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:54:53,470 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:53,470 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:54:53,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:53,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:53,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:53,470 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:54:53,470 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:54:53,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:53,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:53,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:53,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:53,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:53,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:53,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:53,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:54:53,473 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:54:53,473 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:54:53,473 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:54:53,473 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:54:53,473 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:54:53,473 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:54:53,474 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:53,474 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:54:53,474 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:53,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:53,474 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:53,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:53,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:53,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:53,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:53,475 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:54:53,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:54:53,475 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:53,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:53,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:53,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:53,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:53,476 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:53,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:53,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:53,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:53,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:53,478 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:53,478 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:54:53,478 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:53,479 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:53,479 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:53,479 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:53,479 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:53,479 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:53,479 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:54:53,479 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:54:53,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:54:53,479 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:54:53,479 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:54:53,479 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:54:53,479 DEBUG [Thread-1736] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:54:53,479 DEBUG [Thread-1736] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:54:53,479 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:54:53,479 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:54:53,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:54:53,579 DEBUG [Thread-1736] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:54:53,580 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:54:53,581 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:54:53,582 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:54:53,582 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:54:53,583 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:54:53,583 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:54:53,583 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:54:53,583 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:54:53,583 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:53,583 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:53,584 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:54:53,584 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:53,584 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:54:53,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:54:53,585 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:53,585 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:54:53,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:53,585 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:53,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:54:53,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:53,585 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:54:53,586 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:54:53,586 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:54:53,586 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:54:53,586 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:54:53,586 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:54:53,586 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:54:53,586 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:54:53,586 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. 2023-04-16 00:54:53,586 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. started... 2023-04-16 00:54:53,586 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. as already flushing 2023-04-16 00:54:53,586 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. 2023-04-16 00:54:53,586 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:54:53,586 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:54:53,586 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:53,586 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:54:53,586 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:54:53,589 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:53,589 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:53,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:53,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:53,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:53,589 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:53,589 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:54:53,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:53,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:53,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:53,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:53,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:53,591 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:54:53,591 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:54:53,591 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@45c32cc8[Count = 0] remaining members to acquire global barrier 2023-04-16 00:54:53,591 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:53,592 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:53,592 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:53,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:53,592 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:54:53,592 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:54:53,592 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:54:53,592 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:53,592 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:54:53,593 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:54:53,593 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:54:53,593 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:54:53,593 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:53,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:53,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:53,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:53,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:53,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:53,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:53,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:53,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:53,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:53,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:53,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:54:53,596 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:54:53,596 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:54:53,596 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:54:53,596 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:54:53,596 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:54:53,596 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:54:53,601 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:53,601 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:53,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:53,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:53,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:53,601 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:53,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:53,601 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:54:53,602 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:54:53,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:54:53,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:53,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:53,602 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:53,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:53,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:53,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:53,603 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:53,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:53,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:53,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:53,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:53,605 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:53,605 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:54:53,605 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:53,605 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:54:53,605 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:54:53,605 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:54:53,605 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:54:53,605 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:54:53,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:54:53,605 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:53,605 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:53,606 DEBUG [Thread-1736] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:54:53,605 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:54:53,606 DEBUG [Thread-1736] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:54:53,606 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:53,606 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:54:53,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:54:53,606 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:53,606 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:53,654 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:53,654 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58292 deadline: 1681606553653, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:53,683 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=162 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/.tmp/B/d2b8e5a9229d4239b3501a67e0bf4653 2023-04-16 00:54:53,706 DEBUG [Thread-1736] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:54:53,706 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:54:53,708 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:54:53,709 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:54:53,709 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:54:53,709 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:54:53,709 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:54:53,709 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:54:53,709 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:54:53,709 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:53,709 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:53,710 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:54:53,710 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:54:53,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:54:53,710 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:53,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:53,711 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:53,711 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:54:53,711 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:53,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:54:53,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:53,711 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:54:53,712 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:54:53,712 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:54:53,712 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:54:53,712 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:54:53,712 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:54:53,712 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:54:53,712 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:54:53,712 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. 2023-04-16 00:54:53,712 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. started... 2023-04-16 00:54:53,712 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. as already flushing 2023-04-16 00:54:53,712 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. 2023-04-16 00:54:53,712 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:54:53,712 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:54:53,712 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:53,712 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:54:53,712 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:54:53,714 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:53,714 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:53,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:53,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:53,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:53,714 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:53,714 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:54:53,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:53,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:53,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:53,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:53,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:53,715 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:54:53,715 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2a8a0c36[Count = 0] remaining members to acquire global barrier 2023-04-16 00:54:53,715 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:54:53,716 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:53,718 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:53,718 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:53,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:53,718 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:54:53,718 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:54:53,718 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:54:53,718 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:53,718 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:54:53,720 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:53,720 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:54:53,720 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:54:53,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:53,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:53,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:53,720 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:54:53,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:53,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:53,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:53,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:53,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:53,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:53,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:53,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:54:53,723 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:54:53,723 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:54:53,723 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:54:53,723 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:54:53,723 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:54:53,723 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:54:53,724 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:53,724 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:53,724 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:53,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:53,724 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:54:53,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:53,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:53,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:53,724 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:54:53,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:54:53,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:53,725 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:53,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:53,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:53,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:53,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:53,726 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:53,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:53,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:53,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:53,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:53,728 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:53,728 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:53,728 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:54:53,728 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:54:53,728 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:53,728 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:54:53,728 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:54:53,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:54:53,728 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:54:53,728 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:54:53,728 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:53,728 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:53,728 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:53,728 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:53,728 DEBUG [Thread-1736] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:54:53,728 DEBUG [Thread-1736] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:54:53,728 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:54:53,729 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:54:53,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:54:53,829 DEBUG [Thread-1736] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:54:53,829 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:54:53,831 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:54:53,831 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:54:53,832 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:54:53,832 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:54:53,832 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:54:53,832 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:54:53,832 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:54:53,832 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:53,832 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:53,835 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:53,835 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:54:53,835 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:54:53,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:54:53,835 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:53,835 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:54:53,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:53,836 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:53,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:54:53,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:53,836 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:54:53,836 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:54:53,836 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:54:53,836 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:54:53,836 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:54:53,836 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:54:53,837 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:54:53,837 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:54:53,837 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. 2023-04-16 00:54:53,837 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. started... 2023-04-16 00:54:53,837 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. as already flushing 2023-04-16 00:54:53,837 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. 2023-04-16 00:54:53,837 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:54:53,837 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:54:53,837 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:53,837 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:54:53,837 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:54:53,838 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:53,838 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:53,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:53,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:53,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:53,839 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:53,839 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:54:53,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:53,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:53,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:53,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:53,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:53,840 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:54:53,840 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:54:53,840 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@468cb281[Count = 0] remaining members to acquire global barrier 2023-04-16 00:54:53,840 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:53,842 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:53,842 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:53,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:53,842 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:54:53,842 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:53,842 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:54:53,842 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:54:53,842 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:54:53,843 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:54:53,843 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:53,843 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:54:53,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:53,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:53,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:53,843 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:54:53,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:53,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:53,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:53,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:53,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:53,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:53,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:53,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:54:53,846 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:54:53,846 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:54:53,846 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:54:53,846 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:54:53,846 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:54:53,846 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:54:53,848 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:53,848 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:53,848 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:53,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:53,848 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:54:53,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:53,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:53,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:53,849 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:54:53,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:54:53,849 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:53,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:53,850 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:53,852 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:53,852 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:53,852 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:53,852 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:53,852 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:53,852 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:53,852 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:53,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:53,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:53,852 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:54:53,852 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:54:53,852 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:54:53,852 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:54:53,852 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:54:53,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:53,852 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:54:53,853 DEBUG [Thread-1736] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:54:53,853 DEBUG [Thread-1736] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:54:53,853 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:54:53,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:54:53,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:53,853 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:54:53,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:54:53,853 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:54:53,953 DEBUG [Thread-1736] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:54:53,953 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:54:53,955 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:54:53,956 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:54:53,956 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:54:53,956 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:54:53,956 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:54:53,956 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:54:53,956 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:54:53,956 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:53,957 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:53,957 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58292 deadline: 1681606553955, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:53,957 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:53,958 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:54:53,958 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:53,958 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:54:53,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:54:53,958 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:53,958 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:54:53,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:53,959 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:53,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:54:53,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:53,959 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:54:53,959 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:54:53,959 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:54:53,959 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:54:53,959 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:54:53,959 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:54:53,960 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:54:53,960 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:54:53,960 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. 2023-04-16 00:54:53,960 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. started... 2023-04-16 00:54:53,960 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. as already flushing 2023-04-16 00:54:53,960 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. 2023-04-16 00:54:53,960 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:54:53,960 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:54:53,960 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:53,960 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:54:53,960 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:54:53,961 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:53,961 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:53,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:53,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:53,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:53,962 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:53,962 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:54:53,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:53,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:53,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:53,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:53,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:53,963 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:54:53,963 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@39123a40[Count = 0] remaining members to acquire global barrier 2023-04-16 00:54:53,963 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:54:53,963 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:53,964 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:53,964 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:53,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:53,964 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:54:53,964 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:54:53,964 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:54:53,964 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:53,965 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:54:53,967 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:54:53,967 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:53,967 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:54:53,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:53,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:53,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:53,967 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:54:53,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:53,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:53,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:53,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:53,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:53,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:53,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:53,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:54:53,969 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:54:53,969 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:54:53,969 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:54:53,970 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:54:53,970 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:54:53,970 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:54:53,971 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:53,971 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:53,971 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:53,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:53,971 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:54:53,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:53,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:53,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:53,971 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:54:53,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:54:53,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:53,972 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:53,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:53,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:53,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:53,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:53,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:53,973 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:53,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:53,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:53,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:53,977 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:54:53,977 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:53,977 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:54:53,977 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:53,977 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:54:53,977 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:54:53,977 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:54:53,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:54:53,977 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:54:53,977 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:53,977 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:53,977 DEBUG [Thread-1736] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:54:53,978 DEBUG [Thread-1736] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:54:53,977 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:53,978 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:54:53,978 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:53,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:54:53,978 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:54,078 DEBUG [Thread-1736] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:54:54,078 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:54:54,080 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:54:54,081 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:54:54,081 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:54:54,081 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:54:54,081 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:54:54,081 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:54:54,081 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:54:54,081 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:54,081 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:54,083 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:54:54,084 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:54,084 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:54:54,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:54:54,084 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:54,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:54,084 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:54:54,084 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:54,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:54:54,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:54,085 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:54:54,085 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:54:54,085 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:54:54,085 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:54:54,085 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:54:54,085 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:54:54,085 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:54:54,085 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:54:54,085 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. 2023-04-16 00:54:54,085 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. started... 2023-04-16 00:54:54,086 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. as already flushing 2023-04-16 00:54:54,086 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. 2023-04-16 00:54:54,086 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:54:54,086 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:54:54,086 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:54,086 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:54:54,086 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:54:54,087 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:54,087 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:54,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:54,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:54,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:54,088 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:54,088 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:54:54,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:54,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:54,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:54,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:54,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:54,089 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:54:54,089 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@656be4c6[Count = 0] remaining members to acquire global barrier 2023-04-16 00:54:54,089 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:54:54,089 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:54,090 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:54,090 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:54,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:54,090 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:54:54,090 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:54:54,090 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:54:54,091 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:54,091 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:54:54,092 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:54,092 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:54:54,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:54,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:54,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:54,092 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:54:54,092 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:54:54,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:54,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:54,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:54,093 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=162 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/.tmp/C/e1e64f4f28a74feeb7c45332135a1ad7 2023-04-16 00:54:54,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:54,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:54,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:54,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:54,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:54:54,095 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:54:54,095 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:54:54,095 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:54:54,095 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:54:54,095 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:54:54,095 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:54:54,097 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/.tmp/A/395a461575f54884afaddaeac4a27715 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/A/395a461575f54884afaddaeac4a27715 2023-04-16 00:54:54,098 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:54,098 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:54,098 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:54,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:54,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:54,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:54,098 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:54:54,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:54,098 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:54:54,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:54:54,098 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:54,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:54,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:54,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:54,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:54,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:54,100 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:54,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:54,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:54,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:54,101 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/A/395a461575f54884afaddaeac4a27715, entries=150, sequenceid=162, filesize=11.8 K 2023-04-16 00:54:54,102 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/.tmp/B/d2b8e5a9229d4239b3501a67e0bf4653 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/B/d2b8e5a9229d4239b3501a67e0bf4653 2023-04-16 00:54:54,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:54,104 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:54,104 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:54,104 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:54:54,104 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:54,104 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:54:54,104 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:54:54,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:54:54,104 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:54,104 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:54:54,105 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:54:54,104 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:54:54,104 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:54:54,105 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:54,105 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:54,105 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:54,105 DEBUG [Thread-1736] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:54:54,105 DEBUG [Thread-1736] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:54:54,105 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:54:54,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:54:54,106 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/B/d2b8e5a9229d4239b3501a67e0bf4653, entries=150, sequenceid=162, filesize=11.8 K 2023-04-16 00:54:54,107 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/.tmp/C/e1e64f4f28a74feeb7c45332135a1ad7 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/C/e1e64f4f28a74feeb7c45332135a1ad7 2023-04-16 00:54:54,111 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/C/e1e64f4f28a74feeb7c45332135a1ad7, entries=150, sequenceid=162, filesize=11.8 K 2023-04-16 00:54:54,111 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 5dcf6997c5a081edf1d95a6a373c6b08 in 867ms, sequenceid=162, compaction requested=true 2023-04-16 00:54:54,111 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 5dcf6997c5a081edf1d95a6a373c6b08: 2023-04-16 00:54:54,112 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 00:54:54,112 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-16 00:54:54,112 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 00:54:54,112 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-16 00:54:54,112 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 00:54:54,113 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36416 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-16 00:54:54,113 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HStore(1912): 5dcf6997c5a081edf1d95a6a373c6b08/A is initiating minor compaction (all files) 2023-04-16 00:54:54,113 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36416 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-16 00:54:54,113 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HStore(1912): 5dcf6997c5a081edf1d95a6a373c6b08/B is initiating minor compaction (all files) 2023-04-16 00:54:54,113 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 5dcf6997c5a081edf1d95a6a373c6b08/A in TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. 2023-04-16 00:54:54,113 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 5dcf6997c5a081edf1d95a6a373c6b08/B in TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. 2023-04-16 00:54:54,113 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/A/96b5fc95057744639455f4bc8f1fa666, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/A/3ce09f4850a94551ab78a00899f062b9, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/A/395a461575f54884afaddaeac4a27715] into tmpdir=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/.tmp, totalSize=35.6 K 2023-04-16 00:54:54,113 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/B/720dc9d7aa6645c4a9c332a498a4e86d, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/B/c4ca91c939e447aa983f1eebb0c4eccf, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/B/d2b8e5a9229d4239b3501a67e0bf4653] into tmpdir=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/.tmp, totalSize=35.6 K 2023-04-16 00:54:54,113 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.Compactor(207): Compacting 96b5fc95057744639455f4bc8f1fa666, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=137, earliestPutTs=1681606490643 2023-04-16 00:54:54,113 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting 720dc9d7aa6645c4a9c332a498a4e86d, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=137, earliestPutTs=1681606490643 2023-04-16 00:54:54,114 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.Compactor(207): Compacting 3ce09f4850a94551ab78a00899f062b9, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=148, earliestPutTs=1681606491284 2023-04-16 00:54:54,114 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting c4ca91c939e447aa983f1eebb0c4eccf, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=148, earliestPutTs=1681606491284 2023-04-16 00:54:54,114 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.Compactor(207): Compacting 395a461575f54884afaddaeac4a27715, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=162, earliestPutTs=1681606493226 2023-04-16 00:54:54,114 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting d2b8e5a9229d4239b3501a67e0bf4653, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=162, earliestPutTs=1681606493226 2023-04-16 00:54:54,120 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] throttle.PressureAwareThroughputController(145): 5dcf6997c5a081edf1d95a6a373c6b08#A#compaction#252 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-16 00:54:54,120 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] throttle.PressureAwareThroughputController(145): 5dcf6997c5a081edf1d95a6a373c6b08#B#compaction#253 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-16 00:54:54,205 DEBUG [Thread-1736] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:54:54,205 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:54:54,207 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:54:54,209 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:54:54,209 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:54:54,209 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:54:54,209 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:54:54,209 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:54:54,209 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:54:54,209 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:54,209 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:54,211 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:54,211 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:54:54,211 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:54:54,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:54:54,211 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:54,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:54,211 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:54:54,211 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:54,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:54:54,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:54,212 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:54:54,212 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:54:54,212 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:54:54,212 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:54:54,212 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:54:54,212 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:54:54,212 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:54:54,212 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:54:54,212 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. 2023-04-16 00:54:54,213 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. started... 2023-04-16 00:54:54,213 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 5dcf6997c5a081edf1d95a6a373c6b08 3/3 column families, dataSize=134.18 KB heapSize=384.98 KB 2023-04-16 00:54:54,213 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5dcf6997c5a081edf1d95a6a373c6b08, store=A 2023-04-16 00:54:54,213 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 00:54:54,213 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5dcf6997c5a081edf1d95a6a373c6b08, store=B 2023-04-16 00:54:54,213 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 00:54:54,213 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5dcf6997c5a081edf1d95a6a373c6b08, store=C 2023-04-16 00:54:54,213 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 00:54:54,461 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(9158): Flush requested on 5dcf6997c5a081edf1d95a6a373c6b08 2023-04-16 00:54:54,461 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. as already flushing 2023-04-16 00:54:54,481 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:54,481 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58292 deadline: 1681606554479, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:54,532 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/.tmp/A/21e01d66e41248dca66286b70b732221 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/A/21e01d66e41248dca66286b70b732221 2023-04-16 00:54:54,532 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/.tmp/B/7a2472cd8eac43c9a09282a5adbfdfdd as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/B/7a2472cd8eac43c9a09282a5adbfdfdd 2023-04-16 00:54:54,536 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 5dcf6997c5a081edf1d95a6a373c6b08/A of 5dcf6997c5a081edf1d95a6a373c6b08 into 21e01d66e41248dca66286b70b732221(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 00:54:54,536 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5dcf6997c5a081edf1d95a6a373c6b08: 2023-04-16 00:54:54,536 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08., storeName=5dcf6997c5a081edf1d95a6a373c6b08/A, priority=13, startTime=1681606494112; duration=0sec 2023-04-16 00:54:54,537 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 00:54:54,537 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-16 00:54:54,537 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 5dcf6997c5a081edf1d95a6a373c6b08/B of 5dcf6997c5a081edf1d95a6a373c6b08 into 7a2472cd8eac43c9a09282a5adbfdfdd(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 00:54:54,538 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5dcf6997c5a081edf1d95a6a373c6b08: 2023-04-16 00:54:54,538 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08., storeName=5dcf6997c5a081edf1d95a6a373c6b08/B, priority=13, startTime=1681606494112; duration=0sec 2023-04-16 00:54:54,538 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 00:54:54,538 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36416 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-16 00:54:54,538 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HStore(1912): 5dcf6997c5a081edf1d95a6a373c6b08/C is initiating minor compaction (all files) 2023-04-16 00:54:54,538 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 5dcf6997c5a081edf1d95a6a373c6b08/C in TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. 2023-04-16 00:54:54,538 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/C/96e1a998284c456aa0504ba6f47358e3, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/C/b60e05604a59435fb573ac3f216f8a22, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/C/e1e64f4f28a74feeb7c45332135a1ad7] into tmpdir=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/.tmp, totalSize=35.6 K 2023-04-16 00:54:54,539 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.Compactor(207): Compacting 96e1a998284c456aa0504ba6f47358e3, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=137, earliestPutTs=1681606490643 2023-04-16 00:54:54,539 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.Compactor(207): Compacting b60e05604a59435fb573ac3f216f8a22, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=148, earliestPutTs=1681606491284 2023-04-16 00:54:54,539 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.Compactor(207): Compacting e1e64f4f28a74feeb7c45332135a1ad7, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=162, earliestPutTs=1681606493226 2023-04-16 00:54:54,545 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] throttle.PressureAwareThroughputController(145): 5dcf6997c5a081edf1d95a6a373c6b08#C#compaction#255 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-16 00:54:54,552 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/.tmp/C/fb1258db8b4f49b192fc75ab9e45bbbc as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/C/fb1258db8b4f49b192fc75ab9e45bbbc 2023-04-16 00:54:54,556 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 5dcf6997c5a081edf1d95a6a373c6b08/C of 5dcf6997c5a081edf1d95a6a373c6b08 into fb1258db8b4f49b192fc75ab9e45bbbc(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 00:54:54,556 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5dcf6997c5a081edf1d95a6a373c6b08: 2023-04-16 00:54:54,556 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08., storeName=5dcf6997c5a081edf1d95a6a373c6b08/C, priority=13, startTime=1681606494112; duration=0sec 2023-04-16 00:54:54,556 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 00:54:54,583 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:54,583 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58292 deadline: 1681606554582, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:54,621 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=199 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/.tmp/A/594881b111294649973ffe9db9eba08d 2023-04-16 00:54:54,787 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:54,787 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58292 deadline: 1681606554784, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:55,040 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=199 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/.tmp/B/8104b6ceeda84d92a329a2713a104504 2023-04-16 00:54:55,089 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:55,089 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58292 deadline: 1681606555088, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:55,419 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:55,420 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58306 deadline: 1681606555417, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:55,421 DEBUG [Thread-1728] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4132 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08., hostname=jenkins-hbase4.apache.org,34975,1681606419217, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor32.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-16 00:54:55,424 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:55,424 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58336 deadline: 1681606555422, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:55,425 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:55,425 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58282 deadline: 1681606555424, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:55,426 DEBUG [Thread-1732] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4134 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08., hostname=jenkins-hbase4.apache.org,34975,1681606419217, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor32.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-16 00:54:55,427 DEBUG [Thread-1726] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4135 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08., hostname=jenkins-hbase4.apache.org,34975,1681606419217, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor32.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-16 00:54:55,439 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:55,439 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58322 deadline: 1681606555437, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:55,440 DEBUG [Thread-1734] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4150 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08., hostname=jenkins-hbase4.apache.org,34975,1681606419217, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor32.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-16 00:54:55,450 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=199 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/.tmp/C/b57ba0adbfda4c23a83653c5127981c8 2023-04-16 00:54:55,454 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/.tmp/A/594881b111294649973ffe9db9eba08d as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/A/594881b111294649973ffe9db9eba08d 2023-04-16 00:54:55,461 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/A/594881b111294649973ffe9db9eba08d, entries=150, sequenceid=199, filesize=11.8 K 2023-04-16 00:54:55,462 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/.tmp/B/8104b6ceeda84d92a329a2713a104504 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/B/8104b6ceeda84d92a329a2713a104504 2023-04-16 00:54:55,465 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/B/8104b6ceeda84d92a329a2713a104504, entries=150, sequenceid=199, filesize=11.8 K 2023-04-16 00:54:55,466 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/.tmp/C/b57ba0adbfda4c23a83653c5127981c8 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/C/b57ba0adbfda4c23a83653c5127981c8 2023-04-16 00:54:55,473 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/C/b57ba0adbfda4c23a83653c5127981c8, entries=150, sequenceid=199, filesize=11.8 K 2023-04-16 00:54:55,473 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~384.94 KB/394176, currentSize=46.96 KB/48090 for 5dcf6997c5a081edf1d95a6a373c6b08 in 1260ms, sequenceid=199, compaction requested=false 2023-04-16 00:54:55,474 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 5dcf6997c5a081edf1d95a6a373c6b08: 2023-04-16 00:54:55,474 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. 2023-04-16 00:54:55,474 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:54:55,474 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:54:55,474 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:55,474 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:54:55,474 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:54:55,475 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:55,475 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:55,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:55,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:55,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:55,476 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:55,476 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:54:55,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:55,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:55,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:55,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:55,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:55,478 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:54:55,478 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1f7abfe1[Count = 0] remaining members to acquire global barrier 2023-04-16 00:54:55,478 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:54:55,478 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:55,481 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:55,481 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:55,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:55,481 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:54:55,481 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:54:55,481 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:54:55,481 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:55,481 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:54:55,483 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:55,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:55,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:55,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:55,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:55,483 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:54:55,483 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:54:55,483 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:54:55,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:55,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:55,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:55,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:55,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:55,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:55,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:54:55,486 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:54:55,486 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:54:55,486 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:54:55,486 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:54:55,486 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:54:55,486 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:54:55,487 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:55,487 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:54:55,487 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:55,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:55,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:55,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:55,488 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:55,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:55,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:55,488 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:54:55,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:54:55,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:55,488 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:55,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:55,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:55,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:55,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:55,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:55,490 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:55,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:55,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:55,492 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:55,492 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:55,492 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:55,492 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:54:55,492 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:54:55,492 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:54:55,492 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:54:55,492 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:55,492 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:55,492 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:55,492 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:55,492 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:54:55,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:54:55,492 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:54:55,492 DEBUG [Thread-1736] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:54:55,493 DEBUG [Thread-1736] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:54:55,493 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:54:55,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:54:55,593 DEBUG [Thread-1736] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:54:55,593 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:54:55,595 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:54:55,595 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(9158): Flush requested on 5dcf6997c5a081edf1d95a6a373c6b08 2023-04-16 00:54:55,595 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 5dcf6997c5a081edf1d95a6a373c6b08 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-16 00:54:55,596 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5dcf6997c5a081edf1d95a6a373c6b08, store=A 2023-04-16 00:54:55,596 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:54:55,596 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5dcf6997c5a081edf1d95a6a373c6b08, store=B 2023-04-16 00:54:55,596 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:54:55,596 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5dcf6997c5a081edf1d95a6a373c6b08, store=C 2023-04-16 00:54:55,596 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:54:55,596 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:54:55,596 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:54:55,596 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:54:55,596 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:54:55,596 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:54:55,596 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:54:55,597 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:55,597 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:55,599 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:54:55,599 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:54:55,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:54:55,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:55,600 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:55,600 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:55,600 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:55,600 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:54:55,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:54:55,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:55,601 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:54:55,601 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:54:55,602 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:54:55,602 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:54:55,602 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:54:55,602 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:54:55,604 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:54:55,604 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:54:55,604 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. 2023-04-16 00:54:55,604 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. started... 2023-04-16 00:54:55,604 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. as already flushing 2023-04-16 00:54:55,604 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. 2023-04-16 00:54:55,604 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:54:55,605 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:54:55,605 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:55,605 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:54:55,605 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:54:55,606 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:55,606 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:55,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:55,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:55,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:55,606 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:55,606 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:54:55,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:55,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:55,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:55,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:55,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:55,608 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:54:55,608 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3cc10e35[Count = 0] remaining members to acquire global barrier 2023-04-16 00:54:55,608 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:54:55,608 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:55,609 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:55,609 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:55,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:55,610 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:54:55,610 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:54:55,610 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:54:55,610 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:55,610 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:54:55,611 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:54:55,611 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:54:55,611 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:55,611 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:54:55,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:55,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:55,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:55,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:55,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:55,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:55,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:55,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:55,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:55,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:55,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:54:55,614 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:54:55,614 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:54:55,614 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:54:55,614 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:54:55,614 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:54:55,614 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:54:55,617 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:55,617 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:55,617 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:55,617 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:54:55,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:55,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:55,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:55,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:55,618 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:54:55,618 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:55,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:54:55,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:55,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:55,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:55,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:55,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:55,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:55,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:55,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:55,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:55,620 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:55,622 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:54:55,622 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:55,622 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:54:55,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:54:55,622 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:54:55,622 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:54:55,622 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:54:55,622 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:54:55,623 DEBUG [Thread-1736] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:54:55,623 DEBUG [Thread-1736] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:54:55,622 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:55,623 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:54:55,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:54:55,623 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:55,623 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:55,623 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:55,623 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:55,623 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:55,658 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681606495633, max timestamp=1681606495656Num uniques 859; 2023-04-16 00:54:55,658 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-16 00:54:55,658 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681606495633, max timestamp=1681606495656Num uniques 859; 2023-04-16 00:54:55,658 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681606495633, max timestamp=1681606495656Num uniques 859; 2023-04-16 00:54:55,658 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-16 00:54:55,658 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-16 00:54:55,716 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:55,716 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 168 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58292 deadline: 1681606555712, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:55,723 DEBUG [Thread-1736] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:54:55,723 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:54:55,725 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:54:55,726 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:54:55,726 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:54:55,726 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:54:55,726 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:54:55,726 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:54:55,726 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:54:55,726 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:55,727 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:55,731 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:54:55,731 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:55,731 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:54:55,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:54:55,731 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:55,731 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:54:55,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:55,732 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:55,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:54:55,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:55,732 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:54:55,732 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:54:55,732 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:54:55,732 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:54:55,732 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:54:55,732 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:54:55,733 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:54:55,733 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:54:55,733 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. 2023-04-16 00:54:55,733 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. started... 2023-04-16 00:54:55,733 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. as already flushing 2023-04-16 00:54:55,733 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. 2023-04-16 00:54:55,733 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:54:55,733 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:54:55,733 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:55,733 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:54:55,733 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:54:55,735 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:55,735 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:55,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:55,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:55,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:55,736 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:55,736 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:54:55,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:55,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:55,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:55,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:55,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:55,737 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:54:55,737 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1429f3d7[Count = 0] remaining members to acquire global barrier 2023-04-16 00:54:55,737 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:54:55,737 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:55,739 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:55,739 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:55,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:55,739 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:54:55,739 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:55,739 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:54:55,739 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:54:55,739 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:54:55,741 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:54:55,741 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:54:55,741 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:55,741 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:54:55,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:55,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:55,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:55,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:55,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:55,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:55,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:55,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:55,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:55,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:55,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:54:55,743 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:54:55,743 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:54:55,751 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:54:55,751 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:54:55,751 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:54:55,751 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:54:55,753 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:55,753 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:54:55,753 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:55,753 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:55,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:55,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:55,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:55,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:55,753 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:54:55,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:54:55,754 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:55,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:55,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:55,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:55,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:55,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:55,755 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:55,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:55,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:55,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:55,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:55,758 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:55,758 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:54:55,758 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:55,758 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:55,758 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:55,758 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:55,758 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:55,758 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:54:55,758 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:54:55,759 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:54:55,758 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:54:55,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:54:55,758 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:54:55,759 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:54:55,758 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:55,759 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:54:55,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:54:55,759 DEBUG [Thread-1736] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:54:55,759 DEBUG [Thread-1736] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:54:55,819 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:55,819 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 170 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58292 deadline: 1681606555817, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:55,859 DEBUG [Thread-1736] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:54:55,860 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:54:55,861 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:54:55,862 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:54:55,862 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:54:55,862 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:54:55,862 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:54:55,862 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:54:55,862 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:54:55,863 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:55,863 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:55,864 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:54:55,864 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:55,864 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:54:55,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:54:55,865 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:55,865 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:54:55,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:55,865 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:55,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:54:55,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:55,865 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:54:55,866 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:54:55,866 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:54:55,866 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:54:55,866 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:54:55,866 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:54:55,866 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:54:55,866 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. 2023-04-16 00:54:55,866 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. started... 2023-04-16 00:54:55,866 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. as already flushing 2023-04-16 00:54:55,866 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. 2023-04-16 00:54:55,866 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:54:55,866 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:54:55,866 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:55,866 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:54:55,866 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:54:55,868 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:55,868 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:55,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:55,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:55,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:55,868 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:55,868 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:54:55,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:55,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:55,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:55,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:55,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:55,869 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:54:55,869 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:54:55,869 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2e5e6840[Count = 0] remaining members to acquire global barrier 2023-04-16 00:54:55,869 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:55,871 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:55,871 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:55,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:55,871 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:54:55,871 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:54:55,871 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:54:55,871 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:55,871 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:54:55,873 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:55,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:55,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:55,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:55,873 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:54:55,873 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:54:55,873 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:54:55,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:55,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:55,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:55,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:55,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:55,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:55,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:55,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:54:55,875 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:54:55,875 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:54:55,875 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:54:55,875 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:54:55,875 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:54:55,875 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:54:55,877 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:55,877 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:54:55,877 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:55,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:55,877 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:54:55,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:54:55,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:55,878 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:55,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:55,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:55,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:55,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:55,878 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:55,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:55,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:55,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:55,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:55,879 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:55,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:55,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:55,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:55,881 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:54:55,881 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:54:55,881 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:55,881 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:55,882 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:55,882 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:54:55,882 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:54:55,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:54:55,882 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:54:55,882 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:55,882 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:55,882 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:55,882 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:55,882 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:54:55,882 DEBUG [Thread-1736] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:54:55,882 DEBUG [Thread-1736] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:54:55,882 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:54:55,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:54:55,982 DEBUG [Thread-1736] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:54:55,983 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:54:55,984 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:54:55,985 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:54:55,985 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:54:55,985 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:54:55,985 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:54:55,985 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:54:55,985 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:54:55,986 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:55,986 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:55,987 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:54:55,987 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:55,987 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:54:55,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:54:55,987 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:55,987 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:54:55,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:55,988 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:55,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:54:55,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:55,988 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:54:55,988 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:54:55,988 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:54:55,989 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:54:55,989 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:54:55,989 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:54:55,989 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:54:55,989 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:54:55,989 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. 2023-04-16 00:54:55,989 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. started... 2023-04-16 00:54:55,989 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. as already flushing 2023-04-16 00:54:55,989 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. 2023-04-16 00:54:55,989 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:54:55,989 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:54:55,989 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:55,989 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:54:55,989 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:54:55,991 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:55,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:55,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:55,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:55,992 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:55,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:55,992 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:55,992 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:54:55,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:55,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:55,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:55,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:55,993 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:54:55,993 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@64915fdd[Count = 0] remaining members to acquire global barrier 2023-04-16 00:54:55,993 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:54:55,993 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:55,994 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:55,994 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:55,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:55,995 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:54:55,995 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:54:55,995 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:54:55,995 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:55,995 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:54:55,996 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:55,996 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:54:55,996 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:54:55,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:55,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:55,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:55,996 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:54:55,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:55,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:55,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:55,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:55,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:55,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:55,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:55,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:54:55,998 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:54:55,998 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:54:55,999 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:54:55,999 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:54:55,999 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:54:55,999 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:54:56,000 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:56,000 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:56,000 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:56,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:56,000 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:54:56,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:56,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:56,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:56,001 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:54:56,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:54:56,001 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:56,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:56,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:56,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:56,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:56,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:56,002 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:56,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:56,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:56,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:56,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:56,008 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:54:56,008 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:56,008 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:54:56,008 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:54:56,008 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:54:56,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:54:56,008 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:54:56,008 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:56,008 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:56,008 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:56,008 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:56,008 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:56,008 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:56,008 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:54:56,008 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:54:56,009 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:54:56,009 DEBUG [Thread-1736] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:54:56,009 DEBUG [Thread-1736] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:54:56,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:54:56,010 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=213 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/.tmp/A/16d91c09d2e246e289467d4e502b812a 2023-04-16 00:54:56,022 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:56,022 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 172 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58292 deadline: 1681606556020, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:56,109 DEBUG [Thread-1736] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:54:56,109 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:54:56,111 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:54:56,112 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:54:56,112 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:54:56,112 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:54:56,112 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:54:56,112 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:54:56,112 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:54:56,112 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:56,112 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:56,114 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:54:56,114 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:56,114 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:54:56,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:54:56,114 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:56,114 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:54:56,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:56,114 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:56,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:54:56,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:56,115 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:54:56,115 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:54:56,115 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:54:56,115 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:54:56,115 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:54:56,115 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:54:56,115 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:54:56,115 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:54:56,115 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. 2023-04-16 00:54:56,115 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. started... 2023-04-16 00:54:56,116 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. as already flushing 2023-04-16 00:54:56,116 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. 2023-04-16 00:54:56,116 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:54:56,116 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:54:56,116 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:56,116 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:54:56,116 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:54:56,125 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:56,125 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:56,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:56,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:56,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:56,125 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:56,125 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:54:56,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:56,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:56,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:56,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:56,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:56,126 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:54:56,127 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@501545dc[Count = 0] remaining members to acquire global barrier 2023-04-16 00:54:56,127 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:54:56,127 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:56,129 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:56,129 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:56,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:56,129 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:54:56,129 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:54:56,129 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:54:56,129 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:56,129 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:54:56,130 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:54:56,130 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:54:56,130 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:56,130 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:54:56,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:56,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:56,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:56,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:56,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:56,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:56,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:56,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:56,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:56,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:56,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:54:56,133 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:54:56,133 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:54:56,133 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:54:56,133 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:54:56,133 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:54:56,133 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:54:56,135 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:56,135 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:54:56,135 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:56,135 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:56,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:56,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:56,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:56,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:56,135 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:56,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:56,135 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:54:56,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:54:56,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:56,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:56,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:56,137 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:56,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:56,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:56,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:56,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:56,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:56,139 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:54:56,139 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:54:56,139 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:54:56,139 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:54:56,139 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:54:56,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:54:56,139 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:54:56,140 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:54:56,139 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:56,140 DEBUG [Thread-1736] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:54:56,140 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:56,140 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:56,140 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:54:56,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:54:56,140 DEBUG [Thread-1736] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:54:56,140 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:56,140 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:56,140 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:56,140 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:56,240 DEBUG [Thread-1736] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:54:56,241 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:54:56,242 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:54:56,244 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:54:56,244 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:54:56,244 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:54:56,244 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:54:56,244 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:54:56,244 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:54:56,245 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:56,245 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:56,247 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:54:56,247 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:56,247 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:54:56,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:54:56,247 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:56,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:56,247 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:54:56,248 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:56,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:54:56,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:56,248 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:54:56,248 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:54:56,248 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:54:56,248 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:54:56,248 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:54:56,248 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:54:56,249 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:54:56,249 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:54:56,249 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. 2023-04-16 00:54:56,249 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. started... 2023-04-16 00:54:56,249 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. as already flushing 2023-04-16 00:54:56,249 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. 2023-04-16 00:54:56,249 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:54:56,249 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:54:56,249 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:56,249 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:54:56,249 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:54:56,250 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:56,250 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:56,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:56,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:56,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:56,251 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:56,251 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:54:56,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:56,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:56,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:56,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:56,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:56,252 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:54:56,252 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@bf9b342[Count = 0] remaining members to acquire global barrier 2023-04-16 00:54:56,252 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:54:56,252 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:56,253 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:56,253 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:56,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:56,253 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:54:56,253 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:54:56,253 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:54:56,253 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:56,253 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:54:56,255 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:56,255 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:54:56,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:56,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:56,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:56,255 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:54:56,255 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:54:56,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:56,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:56,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:56,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:56,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:56,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:56,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:56,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:54:56,257 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:54:56,257 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:54:56,257 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:54:56,257 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:54:56,257 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:54:56,257 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:54:56,260 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:56,260 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:56,260 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:56,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:56,260 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:54:56,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:56,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:56,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:56,260 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:54:56,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:54:56,260 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:56,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:56,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:56,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:56,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:56,261 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:56,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:56,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:56,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:56,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:56,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:56,263 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:56,263 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:56,263 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:56,263 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:56,263 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:56,263 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:56,263 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:54:56,263 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:54:56,263 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:54:56,263 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:56,263 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:54:56,263 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:54:56,264 DEBUG [Thread-1736] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:54:56,264 DEBUG [Thread-1736] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:54:56,263 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:54:56,264 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:54:56,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:54:56,264 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:54:56,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:54:56,326 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:56,326 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 174 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58292 deadline: 1681606556324, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:56,364 DEBUG [Thread-1736] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:54:56,364 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:54:56,366 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:54:56,367 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:54:56,367 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:54:56,367 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:54:56,367 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:54:56,367 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:54:56,367 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:54:56,367 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:56,367 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:56,368 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:54:56,368 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:56,368 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:54:56,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:54:56,369 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:56,369 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:54:56,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:56,369 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:56,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:54:56,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:56,370 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:54:56,370 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:54:56,370 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:54:56,370 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:54:56,370 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:54:56,370 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:54:56,370 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:54:56,370 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:54:56,370 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. 2023-04-16 00:54:56,370 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. started... 2023-04-16 00:54:56,371 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. as already flushing 2023-04-16 00:54:56,371 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. 2023-04-16 00:54:56,371 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:54:56,371 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:54:56,371 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:56,371 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:54:56,371 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:54:56,372 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:56,372 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:56,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:56,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:56,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:56,372 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:56,372 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:54:56,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:56,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:56,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:56,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:56,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:56,374 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:54:56,374 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5638c615[Count = 0] remaining members to acquire global barrier 2023-04-16 00:54:56,374 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:54:56,374 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:56,376 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:56,376 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:56,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:56,376 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:54:56,376 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:54:56,376 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:56,376 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:54:56,376 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:54:56,377 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:54:56,378 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:54:56,377 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:56,378 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:54:56,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:56,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:56,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:56,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:56,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:56,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:56,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:56,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:56,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:56,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:56,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:54:56,380 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:54:56,380 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:54:56,380 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:54:56,380 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:54:56,380 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:54:56,380 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:54:56,381 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:56,381 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:56,381 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:56,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:56,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:56,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:56,381 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:54:56,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:56,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:56,382 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:56,382 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:54:56,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:54:56,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:56,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:56,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:56,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:56,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:56,383 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:56,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:56,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:56,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:56,386 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:56,386 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:54:56,386 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:56,386 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:54:56,386 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:56,386 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:54:56,386 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:56,386 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:54:56,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:54:56,386 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:54:56,386 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:56,386 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:54:56,386 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:56,386 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:56,386 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:54:56,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:54:56,386 DEBUG [Thread-1736] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:54:56,386 DEBUG [Thread-1736] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:54:56,419 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=213 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/.tmp/B/e07b3cf10a3947bdb6e75b9f55a3abe9 2023-04-16 00:54:56,429 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=213 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/.tmp/C/cd98e72955444c60813ffd6e967016b3 2023-04-16 00:54:56,432 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/.tmp/A/16d91c09d2e246e289467d4e502b812a as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/A/16d91c09d2e246e289467d4e502b812a 2023-04-16 00:54:56,436 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/A/16d91c09d2e246e289467d4e502b812a, entries=200, sequenceid=213, filesize=14.1 K 2023-04-16 00:54:56,436 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/.tmp/B/e07b3cf10a3947bdb6e75b9f55a3abe9 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/B/e07b3cf10a3947bdb6e75b9f55a3abe9 2023-04-16 00:54:56,439 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/B/e07b3cf10a3947bdb6e75b9f55a3abe9, entries=150, sequenceid=213, filesize=11.8 K 2023-04-16 00:54:56,440 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/.tmp/C/cd98e72955444c60813ffd6e967016b3 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/C/cd98e72955444c60813ffd6e967016b3 2023-04-16 00:54:56,443 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/C/cd98e72955444c60813ffd6e967016b3, entries=150, sequenceid=213, filesize=11.8 K 2023-04-16 00:54:56,443 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 5dcf6997c5a081edf1d95a6a373c6b08 in 848ms, sequenceid=213, compaction requested=true 2023-04-16 00:54:56,443 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 5dcf6997c5a081edf1d95a6a373c6b08: 2023-04-16 00:54:56,443 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 00:54:56,444 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 00:54:56,444 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-16 00:54:56,444 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-16 00:54:56,444 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 00:54:56,445 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36518 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-16 00:54:56,445 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 38908 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-16 00:54:56,445 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HStore(1912): 5dcf6997c5a081edf1d95a6a373c6b08/B is initiating minor compaction (all files) 2023-04-16 00:54:56,445 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HStore(1912): 5dcf6997c5a081edf1d95a6a373c6b08/A is initiating minor compaction (all files) 2023-04-16 00:54:56,445 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 5dcf6997c5a081edf1d95a6a373c6b08/B in TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. 2023-04-16 00:54:56,445 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 5dcf6997c5a081edf1d95a6a373c6b08/A in TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. 2023-04-16 00:54:56,445 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/B/7a2472cd8eac43c9a09282a5adbfdfdd, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/B/8104b6ceeda84d92a329a2713a104504, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/B/e07b3cf10a3947bdb6e75b9f55a3abe9] into tmpdir=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/.tmp, totalSize=35.7 K 2023-04-16 00:54:56,445 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/A/21e01d66e41248dca66286b70b732221, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/A/594881b111294649973ffe9db9eba08d, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/A/16d91c09d2e246e289467d4e502b812a] into tmpdir=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/.tmp, totalSize=38.0 K 2023-04-16 00:54:56,445 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting 7a2472cd8eac43c9a09282a5adbfdfdd, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=162, earliestPutTs=1681606493226 2023-04-16 00:54:56,445 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.Compactor(207): Compacting 21e01d66e41248dca66286b70b732221, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=162, earliestPutTs=1681606493226 2023-04-16 00:54:56,446 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting 8104b6ceeda84d92a329a2713a104504, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=199, earliestPutTs=1681606493310 2023-04-16 00:54:56,446 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.Compactor(207): Compacting 594881b111294649973ffe9db9eba08d, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=199, earliestPutTs=1681606493310 2023-04-16 00:54:56,446 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting e07b3cf10a3947bdb6e75b9f55a3abe9, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=213, earliestPutTs=1681606494468 2023-04-16 00:54:56,446 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.Compactor(207): Compacting 16d91c09d2e246e289467d4e502b812a, keycount=200, bloomtype=ROW, size=14.1 K, encoding=NONE, compression=NONE, seqNum=213, earliestPutTs=1681606494468 2023-04-16 00:54:56,452 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] throttle.PressureAwareThroughputController(145): 5dcf6997c5a081edf1d95a6a373c6b08#A#compaction#261 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-16 00:54:56,452 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] throttle.PressureAwareThroughputController(145): 5dcf6997c5a081edf1d95a6a373c6b08#B#compaction#262 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-16 00:54:56,463 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/.tmp/B/5cf2c5d02ae249d7955278274d8aa969 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/B/5cf2c5d02ae249d7955278274d8aa969 2023-04-16 00:54:56,466 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 5dcf6997c5a081edf1d95a6a373c6b08/B of 5dcf6997c5a081edf1d95a6a373c6b08 into 5cf2c5d02ae249d7955278274d8aa969(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 00:54:56,467 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5dcf6997c5a081edf1d95a6a373c6b08: 2023-04-16 00:54:56,467 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08., storeName=5dcf6997c5a081edf1d95a6a373c6b08/B, priority=13, startTime=1681606496444; duration=0sec 2023-04-16 00:54:56,467 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 00:54:56,467 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-16 00:54:56,467 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36518 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-16 00:54:56,467 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HStore(1912): 5dcf6997c5a081edf1d95a6a373c6b08/C is initiating minor compaction (all files) 2023-04-16 00:54:56,467 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 5dcf6997c5a081edf1d95a6a373c6b08/C in TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. 2023-04-16 00:54:56,468 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/C/fb1258db8b4f49b192fc75ab9e45bbbc, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/C/b57ba0adbfda4c23a83653c5127981c8, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/C/cd98e72955444c60813ffd6e967016b3] into tmpdir=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/.tmp, totalSize=35.7 K 2023-04-16 00:54:56,468 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting fb1258db8b4f49b192fc75ab9e45bbbc, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=162, earliestPutTs=1681606493226 2023-04-16 00:54:56,468 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting b57ba0adbfda4c23a83653c5127981c8, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=199, earliestPutTs=1681606493310 2023-04-16 00:54:56,469 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting cd98e72955444c60813ffd6e967016b3, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=213, earliestPutTs=1681606494468 2023-04-16 00:54:56,473 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] throttle.PressureAwareThroughputController(145): 5dcf6997c5a081edf1d95a6a373c6b08#C#compaction#263 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-16 00:54:56,486 DEBUG [Thread-1736] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:54:56,487 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:54:56,488 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:54:56,489 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:54:56,489 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:54:56,489 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:54:56,489 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:54:56,489 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:54:56,489 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:54:56,490 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:56,490 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:56,492 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:54:56,492 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:56,492 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:54:56,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:54:56,492 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:56,492 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:54:56,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:56,492 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:56,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:54:56,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:56,493 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:54:56,493 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:54:56,493 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:54:56,493 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:54:56,493 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:54:56,493 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:54:56,493 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:54:56,493 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:54:56,493 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. 2023-04-16 00:54:56,494 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. started... 2023-04-16 00:54:56,494 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 5dcf6997c5a081edf1d95a6a373c6b08 3/3 column families, dataSize=127.47 KB heapSize=365.06 KB 2023-04-16 00:54:56,494 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5dcf6997c5a081edf1d95a6a373c6b08, store=A 2023-04-16 00:54:56,494 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 00:54:56,494 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5dcf6997c5a081edf1d95a6a373c6b08, store=B 2023-04-16 00:54:56,494 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 00:54:56,494 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5dcf6997c5a081edf1d95a6a373c6b08, store=C 2023-04-16 00:54:56,494 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 00:54:56,831 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(9158): Flush requested on 5dcf6997c5a081edf1d95a6a373c6b08 2023-04-16 00:54:56,831 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. as already flushing 2023-04-16 00:54:56,854 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:56,854 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 184 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58292 deadline: 1681606556852, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:56,864 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/.tmp/A/3b716faa19c94eed94906c9cb3cc426e as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/A/3b716faa19c94eed94906c9cb3cc426e 2023-04-16 00:54:56,868 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 5dcf6997c5a081edf1d95a6a373c6b08/A of 5dcf6997c5a081edf1d95a6a373c6b08 into 3b716faa19c94eed94906c9cb3cc426e(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 00:54:56,868 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5dcf6997c5a081edf1d95a6a373c6b08: 2023-04-16 00:54:56,868 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08., storeName=5dcf6997c5a081edf1d95a6a373c6b08/A, priority=13, startTime=1681606496443; duration=0sec 2023-04-16 00:54:56,868 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 00:54:56,881 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/.tmp/C/817e1c997c3c4ca789942f7f4cc9bc2c as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/C/817e1c997c3c4ca789942f7f4cc9bc2c 2023-04-16 00:54:56,885 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 5dcf6997c5a081edf1d95a6a373c6b08/C of 5dcf6997c5a081edf1d95a6a373c6b08 into 817e1c997c3c4ca789942f7f4cc9bc2c(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 00:54:56,885 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5dcf6997c5a081edf1d95a6a373c6b08: 2023-04-16 00:54:56,885 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08., storeName=5dcf6997c5a081edf1d95a6a373c6b08/C, priority=13, startTime=1681606496444; duration=0sec 2023-04-16 00:54:56,885 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 00:54:56,902 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=250 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/.tmp/A/493a39147c7e4c09a8e6b6cc2f778a71 2023-04-16 00:54:56,956 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:56,956 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 186 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58292 deadline: 1681606556955, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:57,159 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:57,160 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 188 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58292 deadline: 1681606557158, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:57,313 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=250 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/.tmp/B/ad79c85f7ba949e189083c0e3c80155f 2023-04-16 00:54:57,323 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=250 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/.tmp/C/559f4f6d943e4ae887accca147a58da9 2023-04-16 00:54:57,327 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/.tmp/A/493a39147c7e4c09a8e6b6cc2f778a71 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/A/493a39147c7e4c09a8e6b6cc2f778a71 2023-04-16 00:54:57,330 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/A/493a39147c7e4c09a8e6b6cc2f778a71, entries=150, sequenceid=250, filesize=11.8 K 2023-04-16 00:54:57,331 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/.tmp/B/ad79c85f7ba949e189083c0e3c80155f as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/B/ad79c85f7ba949e189083c0e3c80155f 2023-04-16 00:54:57,334 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/B/ad79c85f7ba949e189083c0e3c80155f, entries=150, sequenceid=250, filesize=11.8 K 2023-04-16 00:54:57,334 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/.tmp/C/559f4f6d943e4ae887accca147a58da9 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/C/559f4f6d943e4ae887accca147a58da9 2023-04-16 00:54:57,337 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/C/559f4f6d943e4ae887accca147a58da9, entries=150, sequenceid=250, filesize=11.8 K 2023-04-16 00:54:57,338 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~365.02 KB/373776, currentSize=53.67 KB/54960 for 5dcf6997c5a081edf1d95a6a373c6b08 in 844ms, sequenceid=250, compaction requested=false 2023-04-16 00:54:57,338 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 5dcf6997c5a081edf1d95a6a373c6b08: 2023-04-16 00:54:57,338 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. 2023-04-16 00:54:57,338 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:54:57,338 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:54:57,338 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:57,338 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:54:57,338 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:54:57,340 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:57,340 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:57,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:57,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:57,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:57,340 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:57,340 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:54:57,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:57,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:57,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:57,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:57,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:57,342 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:54:57,342 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@38b904b8[Count = 0] remaining members to acquire global barrier 2023-04-16 00:54:57,342 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:54:57,342 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:57,343 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:57,343 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:57,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:57,343 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:54:57,343 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:54:57,343 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:54:57,344 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:57,344 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:54:57,348 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:54:57,348 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:54:57,348 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:57,348 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:54:57,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:57,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:57,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:57,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:57,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:57,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:57,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:57,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:57,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:57,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:57,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:54:57,351 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:54:57,351 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:54:57,351 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:54:57,351 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:54:57,351 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:54:57,351 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:54:57,352 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:57,352 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:57,352 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:57,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:57,352 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:54:57,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:57,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:57,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:57,353 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:54:57,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:54:57,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:57,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:57,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:57,354 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:57,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:57,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:57,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:57,355 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:57,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:57,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:57,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:57,357 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:57,357 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:54:57,357 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:54:57,357 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:57,357 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:54:57,357 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:54:57,357 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:54:57,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:54:57,357 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:57,358 DEBUG [Thread-1736] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:54:57,358 DEBUG [Thread-1736] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:54:57,357 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:54:57,358 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:57,358 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:57,358 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:57,358 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:57,358 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:54:57,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:54:57,458 DEBUG [Thread-1736] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:54:57,458 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:54:57,461 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:54:57,461 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:54:57,462 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:54:57,462 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:54:57,462 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:54:57,462 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:54:57,462 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:54:57,462 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:57,462 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:57,463 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(9158): Flush requested on 5dcf6997c5a081edf1d95a6a373c6b08 2023-04-16 00:54:57,463 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 5dcf6997c5a081edf1d95a6a373c6b08 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-16 00:54:57,463 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5dcf6997c5a081edf1d95a6a373c6b08, store=A 2023-04-16 00:54:57,463 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:54:57,463 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5dcf6997c5a081edf1d95a6a373c6b08, store=B 2023-04-16 00:54:57,463 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:54:57,463 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5dcf6997c5a081edf1d95a6a373c6b08, store=C 2023-04-16 00:54:57,463 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:54:57,464 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:54:57,464 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:57,464 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:54:57,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:54:57,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:57,465 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:57,465 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:57,465 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:54:57,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:54:57,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:57,466 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:54:57,466 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:54:57,466 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:54:57,466 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:54:57,466 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:54:57,466 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:54:57,466 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:54:57,466 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:54:57,466 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. 2023-04-16 00:54:57,466 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. started... 2023-04-16 00:54:57,466 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. as already flushing 2023-04-16 00:54:57,466 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. 2023-04-16 00:54:57,466 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:54:57,466 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:54:57,466 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:57,467 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:54:57,467 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:54:57,468 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:57,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:57,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:57,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:57,468 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:57,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:57,469 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:57,469 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:54:57,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:57,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:57,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:57,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:57,471 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:54:57,471 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@42789dd6[Count = 0] remaining members to acquire global barrier 2023-04-16 00:54:57,471 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:54:57,471 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:57,472 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=264 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/.tmp/A/0cfe3148d92049a1914960bb03abdc6c 2023-04-16 00:54:57,473 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:57,473 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:57,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:57,473 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:54:57,473 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:54:57,473 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:54:57,473 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:57,473 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:54:57,474 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:54:57,474 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:54:57,474 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:57,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:57,474 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:54:57,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:57,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:57,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:57,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:57,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:57,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:57,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:57,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:57,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:57,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:54:57,478 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:54:57,478 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:54:57,478 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:54:57,478 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:54:57,478 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:54:57,478 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:54:57,481 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:57,481 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:57,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:57,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:57,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:57,481 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:57,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:57,481 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:54:57,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:57,482 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:54:57,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:54:57,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:57,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:57,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:57,485 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=264 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/.tmp/B/137031767256439991bf77d0ea6987a2 2023-04-16 00:54:57,485 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:57,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:57,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:57,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:57,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:57,488 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:57,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:57,491 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:54:57,491 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:57,491 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:54:57,491 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:57,491 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:57,491 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:57,491 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:57,491 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:57,491 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:57,491 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:54:57,491 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:54:57,491 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:54:57,491 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:54:57,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:54:57,491 DEBUG [Thread-1736] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:54:57,491 DEBUG [Thread-1736] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:54:57,492 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:54:57,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:54:57,496 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=264 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/.tmp/C/5d4247d06e8c4949a2c18ec9add67da4 2023-04-16 00:54:57,499 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/.tmp/A/0cfe3148d92049a1914960bb03abdc6c as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/A/0cfe3148d92049a1914960bb03abdc6c 2023-04-16 00:54:57,509 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/A/0cfe3148d92049a1914960bb03abdc6c, entries=150, sequenceid=264, filesize=11.9 K 2023-04-16 00:54:57,510 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/.tmp/B/137031767256439991bf77d0ea6987a2 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/B/137031767256439991bf77d0ea6987a2 2023-04-16 00:54:57,513 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/B/137031767256439991bf77d0ea6987a2, entries=150, sequenceid=264, filesize=11.9 K 2023-04-16 00:54:57,513 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/.tmp/C/5d4247d06e8c4949a2c18ec9add67da4 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/C/5d4247d06e8c4949a2c18ec9add67da4 2023-04-16 00:54:57,516 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/C/5d4247d06e8c4949a2c18ec9add67da4, entries=150, sequenceid=264, filesize=11.9 K 2023-04-16 00:54:57,517 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=107.34 KB/109920 for 5dcf6997c5a081edf1d95a6a373c6b08 in 54ms, sequenceid=264, compaction requested=true 2023-04-16 00:54:57,517 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 5dcf6997c5a081edf1d95a6a373c6b08: 2023-04-16 00:54:57,517 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 00:54:57,517 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-16 00:54:57,517 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-16 00:54:57,517 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 00:54:57,518 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 00:54:57,518 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36770 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-16 00:54:57,518 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36770 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-16 00:54:57,518 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HStore(1912): 5dcf6997c5a081edf1d95a6a373c6b08/A is initiating minor compaction (all files) 2023-04-16 00:54:57,518 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HStore(1912): 5dcf6997c5a081edf1d95a6a373c6b08/B is initiating minor compaction (all files) 2023-04-16 00:54:57,519 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 5dcf6997c5a081edf1d95a6a373c6b08/A in TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. 2023-04-16 00:54:57,519 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 5dcf6997c5a081edf1d95a6a373c6b08/B in TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. 2023-04-16 00:54:57,519 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/A/3b716faa19c94eed94906c9cb3cc426e, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/A/493a39147c7e4c09a8e6b6cc2f778a71, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/A/0cfe3148d92049a1914960bb03abdc6c] into tmpdir=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/.tmp, totalSize=35.9 K 2023-04-16 00:54:57,519 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/B/5cf2c5d02ae249d7955278274d8aa969, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/B/ad79c85f7ba949e189083c0e3c80155f, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/B/137031767256439991bf77d0ea6987a2] into tmpdir=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/.tmp, totalSize=35.9 K 2023-04-16 00:54:57,519 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.Compactor(207): Compacting 3b716faa19c94eed94906c9cb3cc426e, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=213, earliestPutTs=1681606494468 2023-04-16 00:54:57,519 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting 5cf2c5d02ae249d7955278274d8aa969, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=213, earliestPutTs=1681606494468 2023-04-16 00:54:57,519 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.Compactor(207): Compacting 493a39147c7e4c09a8e6b6cc2f778a71, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=250, earliestPutTs=1681606495705 2023-04-16 00:54:57,520 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting ad79c85f7ba949e189083c0e3c80155f, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=250, earliestPutTs=1681606495705 2023-04-16 00:54:57,520 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.Compactor(207): Compacting 0cfe3148d92049a1914960bb03abdc6c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=264, earliestPutTs=1681606496847 2023-04-16 00:54:57,520 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(9158): Flush requested on 5dcf6997c5a081edf1d95a6a373c6b08 2023-04-16 00:54:57,520 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 5dcf6997c5a081edf1d95a6a373c6b08 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-16 00:54:57,521 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5dcf6997c5a081edf1d95a6a373c6b08, store=A 2023-04-16 00:54:57,521 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:54:57,521 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting 137031767256439991bf77d0ea6987a2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=264, earliestPutTs=1681606496847 2023-04-16 00:54:57,521 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5dcf6997c5a081edf1d95a6a373c6b08, store=B 2023-04-16 00:54:57,521 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:54:57,521 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5dcf6997c5a081edf1d95a6a373c6b08, store=C 2023-04-16 00:54:57,521 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:54:57,529 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] throttle.PressureAwareThroughputController(145): 5dcf6997c5a081edf1d95a6a373c6b08#A#compaction#271 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-16 00:54:57,530 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] throttle.PressureAwareThroughputController(145): 5dcf6997c5a081edf1d95a6a373c6b08#B#compaction#272 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-16 00:54:57,543 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=284 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/.tmp/A/3b6431bd48ec4e368d2fb70f1a3ee38d 2023-04-16 00:54:57,562 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:57,563 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 217 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58292 deadline: 1681606557559, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:57,591 DEBUG [Thread-1736] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:54:57,592 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:54:57,593 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:54:57,595 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:54:57,595 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:54:57,596 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:54:57,596 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:54:57,596 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:54:57,596 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:54:57,596 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:57,596 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:57,597 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:57,597 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:54:57,598 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:54:57,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:54:57,598 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:57,598 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:54:57,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:57,598 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:57,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:54:57,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:57,598 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:54:57,599 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:54:57,599 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:54:57,599 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:54:57,599 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:54:57,599 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:54:57,599 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:54:57,599 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:54:57,599 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. 2023-04-16 00:54:57,599 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. started... 2023-04-16 00:54:57,599 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. as already flushing 2023-04-16 00:54:57,599 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. 2023-04-16 00:54:57,599 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:54:57,599 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:54:57,599 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:57,599 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:54:57,599 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:54:57,601 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:57,601 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:57,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:57,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:57,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:57,601 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:57,601 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:54:57,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:57,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:57,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:57,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:57,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:57,603 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:54:57,603 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4dd4578[Count = 0] remaining members to acquire global barrier 2023-04-16 00:54:57,603 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:54:57,603 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:57,604 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:57,604 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:57,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:57,605 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:54:57,605 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:54:57,605 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:54:57,605 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:57,605 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:54:57,606 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:54:57,606 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:54:57,606 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:57,606 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:54:57,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:57,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:57,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:57,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:57,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:57,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:57,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:57,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:57,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:57,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:57,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:54:57,608 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:54:57,608 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:54:57,608 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:54:57,609 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:54:57,609 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:54:57,609 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:54:57,610 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:57,610 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:54:57,610 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:57,610 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:57,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:57,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:57,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:57,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:57,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:57,610 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:54:57,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:54:57,611 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:57,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:57,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:57,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:57,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:57,612 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:57,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:57,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:57,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:57,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:57,614 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:57,614 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:57,614 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:57,614 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:57,614 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:54:57,614 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:54:57,614 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:57,614 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:54:57,614 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:54:57,614 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:57,614 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:57,614 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:54:57,614 DEBUG [Thread-1736] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:54:57,614 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:54:57,614 DEBUG [Thread-1736] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:54:57,614 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:54:57,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:54:57,614 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:54:57,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:54:57,665 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:57,665 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 219 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58292 deadline: 1681606557663, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:57,714 DEBUG [Thread-1736] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:54:57,715 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:54:57,716 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:54:57,717 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:54:57,717 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:54:57,717 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:54:57,717 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:54:57,717 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:54:57,717 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:54:57,718 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:57,718 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:57,720 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:57,720 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:54:57,720 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:54:57,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:54:57,720 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:57,720 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:54:57,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:57,720 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:57,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:54:57,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:57,721 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:54:57,721 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:54:57,721 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:54:57,721 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:54:57,721 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:54:57,721 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:54:57,721 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:54:57,721 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:54:57,721 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. 2023-04-16 00:54:57,721 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. started... 2023-04-16 00:54:57,721 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. as already flushing 2023-04-16 00:54:57,721 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. 2023-04-16 00:54:57,721 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:54:57,721 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:54:57,721 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:57,721 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:54:57,722 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:54:57,723 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:57,723 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:57,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:57,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:57,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:57,723 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:57,723 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:54:57,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:57,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:57,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:57,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:57,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:57,724 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:54:57,724 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:54:57,724 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@14c52398[Count = 0] remaining members to acquire global barrier 2023-04-16 00:54:57,724 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:57,725 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:57,725 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:57,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:57,726 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:54:57,726 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:54:57,726 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:54:57,726 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:57,726 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:54:57,727 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:57,727 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:54:57,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:57,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:57,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:57,727 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:54:57,727 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:54:57,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:57,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:57,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:57,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:57,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:57,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:57,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:57,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:54:57,729 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:54:57,729 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:54:57,729 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:54:57,729 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:54:57,729 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:54:57,729 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:54:57,734 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:57,734 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:57,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:57,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:57,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:57,734 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:57,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:57,734 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:54:57,734 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:57,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:57,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:57,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:57,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:57,736 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:57,736 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:54:57,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:54:57,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:57,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:57,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:57,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:57,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:57,741 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:57,741 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:54:57,741 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:57,742 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:54:57,742 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:57,742 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:54:57,742 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:57,742 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:54:57,742 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:54:57,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:54:57,742 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:54:57,742 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:57,742 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:57,742 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:54:57,742 DEBUG [Thread-1736] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:54:57,742 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:57,742 DEBUG [Thread-1736] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:54:57,742 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:54:57,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:54:57,842 DEBUG [Thread-1736] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:54:57,843 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:54:57,845 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:54:57,851 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:54:57,851 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:54:57,851 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:54:57,851 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:54:57,851 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:54:57,851 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:54:57,852 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:57,852 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:57,853 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:54:57,853 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:57,853 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:54:57,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:54:57,853 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:57,853 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:54:57,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:57,854 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:57,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:54:57,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:57,854 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:54:57,856 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:54:57,856 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:54:57,856 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:54:57,856 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:54:57,856 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:54:57,856 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:54:57,856 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:54:57,856 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. 2023-04-16 00:54:57,856 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. started... 2023-04-16 00:54:57,856 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. as already flushing 2023-04-16 00:54:57,856 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. 2023-04-16 00:54:57,856 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:54:57,856 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:54:57,856 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:57,857 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:54:57,857 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:54:57,859 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:57,859 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:57,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:57,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:57,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:57,860 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:57,860 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:54:57,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:57,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:57,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:57,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:57,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:57,861 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:54:57,861 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@64a9797c[Count = 0] remaining members to acquire global barrier 2023-04-16 00:54:57,861 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:54:57,861 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:57,863 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:57,863 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:57,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:57,863 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:54:57,863 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:57,863 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:54:57,863 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:54:57,863 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:54:57,864 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:57,864 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:54:57,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:57,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:57,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:57,864 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:54:57,864 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:54:57,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:57,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:57,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:57,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:57,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:57,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:57,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:57,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:54:57,866 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:54:57,866 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:54:57,867 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:54:57,867 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:54:57,867 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:54:57,867 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:54:57,868 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:57,868 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:57,868 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:57,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:57,868 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:54:57,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:57,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:57,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:57,869 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:54:57,869 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:57,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:54:57,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:57,869 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:57,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:57,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:57,869 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 221 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58292 deadline: 1681606557867, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:57,870 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:57,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:57,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:57,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:57,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:57,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:57,872 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:57,873 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:57,873 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:57,873 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:57,873 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:54:57,873 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:54:57,873 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:57,873 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:57,873 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:57,873 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:54:57,873 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:54:57,873 DEBUG [Thread-1736] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:54:57,873 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:54:57,873 DEBUG [Thread-1736] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:54:57,873 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:54:57,873 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:54:57,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:54:57,874 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:54:57,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:54:57,954 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=284 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/.tmp/B/11e14a78f35741d385ec8ab39d209a8e 2023-04-16 00:54:57,955 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/.tmp/A/ea063ba4012d48b7ac186c0b20fc87ea as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/A/ea063ba4012d48b7ac186c0b20fc87ea 2023-04-16 00:54:57,956 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/.tmp/B/b69c4c09edf74bb19ff8957d7ddfb6f3 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/B/b69c4c09edf74bb19ff8957d7ddfb6f3 2023-04-16 00:54:57,960 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 5dcf6997c5a081edf1d95a6a373c6b08/A of 5dcf6997c5a081edf1d95a6a373c6b08 into ea063ba4012d48b7ac186c0b20fc87ea(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 00:54:57,960 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5dcf6997c5a081edf1d95a6a373c6b08: 2023-04-16 00:54:57,960 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08., storeName=5dcf6997c5a081edf1d95a6a373c6b08/A, priority=13, startTime=1681606497517; duration=0sec 2023-04-16 00:54:57,961 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 00:54:57,961 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-16 00:54:57,961 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 5dcf6997c5a081edf1d95a6a373c6b08/B of 5dcf6997c5a081edf1d95a6a373c6b08 into b69c4c09edf74bb19ff8957d7ddfb6f3(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 00:54:57,961 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5dcf6997c5a081edf1d95a6a373c6b08: 2023-04-16 00:54:57,961 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08., storeName=5dcf6997c5a081edf1d95a6a373c6b08/B, priority=13, startTime=1681606497517; duration=0sec 2023-04-16 00:54:57,961 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 00:54:57,962 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36770 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-16 00:54:57,962 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HStore(1912): 5dcf6997c5a081edf1d95a6a373c6b08/C is initiating minor compaction (all files) 2023-04-16 00:54:57,962 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 5dcf6997c5a081edf1d95a6a373c6b08/C in TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. 2023-04-16 00:54:57,962 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/C/817e1c997c3c4ca789942f7f4cc9bc2c, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/C/559f4f6d943e4ae887accca147a58da9, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/C/5d4247d06e8c4949a2c18ec9add67da4] into tmpdir=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/.tmp, totalSize=35.9 K 2023-04-16 00:54:57,962 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.Compactor(207): Compacting 817e1c997c3c4ca789942f7f4cc9bc2c, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=213, earliestPutTs=1681606494468 2023-04-16 00:54:57,963 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.Compactor(207): Compacting 559f4f6d943e4ae887accca147a58da9, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=250, earliestPutTs=1681606495705 2023-04-16 00:54:57,963 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.Compactor(207): Compacting 5d4247d06e8c4949a2c18ec9add67da4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=264, earliestPutTs=1681606496847 2023-04-16 00:54:57,968 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=284 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/.tmp/C/78c60dd18be44cb7acca58b59ac7efcd 2023-04-16 00:54:57,970 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] throttle.PressureAwareThroughputController(145): 5dcf6997c5a081edf1d95a6a373c6b08#C#compaction#275 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-16 00:54:57,973 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/.tmp/A/3b6431bd48ec4e368d2fb70f1a3ee38d as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/A/3b6431bd48ec4e368d2fb70f1a3ee38d 2023-04-16 00:54:57,973 DEBUG [Thread-1736] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:54:57,974 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:54:57,975 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:54:57,982 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/A/3b6431bd48ec4e368d2fb70f1a3ee38d, entries=200, sequenceid=284, filesize=14.3 K 2023-04-16 00:54:57,982 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/.tmp/B/11e14a78f35741d385ec8ab39d209a8e as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/B/11e14a78f35741d385ec8ab39d209a8e 2023-04-16 00:54:57,986 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:54:57,986 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:54:57,986 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:54:57,986 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:54:57,986 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:54:57,986 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:54:57,986 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:57,986 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:57,990 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:57,990 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:54:57,990 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:54:57,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:54:57,990 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:57,990 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:54:57,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:57,991 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:57,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:54:57,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:57,991 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:54:57,992 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:54:57,992 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:54:57,992 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:54:57,992 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:54:57,992 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:54:57,993 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/B/11e14a78f35741d385ec8ab39d209a8e, entries=150, sequenceid=284, filesize=11.9 K 2023-04-16 00:54:57,993 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:54:57,993 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:54:57,993 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. 2023-04-16 00:54:57,993 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. started... 2023-04-16 00:54:57,993 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. as already flushing 2023-04-16 00:54:57,993 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. 2023-04-16 00:54:57,993 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:54:57,993 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:54:57,993 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:57,993 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:54:57,993 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:54:57,994 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/.tmp/C/78c60dd18be44cb7acca58b59ac7efcd as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/C/78c60dd18be44cb7acca58b59ac7efcd 2023-04-16 00:54:57,995 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:57,995 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:57,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:57,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:57,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:57,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:57,995 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:57,996 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:54:57,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:57,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:57,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:57,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:57,997 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:54:57,997 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:54:57,997 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6239de12[Count = 0] remaining members to acquire global barrier 2023-04-16 00:54:57,997 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:57,998 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/.tmp/C/1fa18240d3e6447ba26dc86aa6049938 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/C/1fa18240d3e6447ba26dc86aa6049938 2023-04-16 00:54:57,999 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:57,999 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:57,999 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/C/78c60dd18be44cb7acca58b59ac7efcd, entries=150, sequenceid=284, filesize=11.9 K 2023-04-16 00:54:57,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:57,999 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:57,999 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:54:57,999 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:54:57,999 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:54:57,999 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:54:57,999 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for 5dcf6997c5a081edf1d95a6a373c6b08 in 479ms, sequenceid=284, compaction requested=false 2023-04-16 00:54:58,000 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 5dcf6997c5a081edf1d95a6a373c6b08: 2023-04-16 00:54:58,001 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:58,001 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:54:58,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:58,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:58,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:58,002 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:54:58,002 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:54:58,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:58,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:58,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:58,003 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 5dcf6997c5a081edf1d95a6a373c6b08/C of 5dcf6997c5a081edf1d95a6a373c6b08 into 1fa18240d3e6447ba26dc86aa6049938(size=12.5 K), total size for store is 24.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 00:54:58,003 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5dcf6997c5a081edf1d95a6a373c6b08: 2023-04-16 00:54:58,003 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08., storeName=5dcf6997c5a081edf1d95a6a373c6b08/C, priority=13, startTime=1681606497518; duration=0sec 2023-04-16 00:54:58,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:58,003 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 00:54:58,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:58,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:58,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:58,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:54:58,004 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:54:58,004 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:54:58,004 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:54:58,004 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:54:58,004 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:54:58,004 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:54:58,006 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:58,006 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:58,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:58,006 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:58,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:58,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:58,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:58,006 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:54:58,006 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:54:58,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:54:58,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:58,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:58,007 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:58,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:58,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:58,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:58,008 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:58,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:58,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:58,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:58,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:58,010 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:58,010 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:54:58,010 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:58,010 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:58,010 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:58,010 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:58,010 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:58,010 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:58,010 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:54:58,010 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:54:58,010 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:54:58,010 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:54:58,010 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:54:58,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:54:58,010 DEBUG [Thread-1736] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:54:58,011 DEBUG [Thread-1736] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:54:58,011 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:54:58,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:54:58,011 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:54:58,111 DEBUG [Thread-1736] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:54:58,111 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:54:58,113 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:54:58,114 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:54:58,114 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:54:58,114 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:54:58,114 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:54:58,114 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:54:58,114 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:54:58,115 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:58,115 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:58,116 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:54:58,116 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:54:58,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:54:58,116 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:58,116 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:58,116 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:54:58,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:58,117 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:58,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:54:58,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:58,117 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:54:58,117 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:54:58,118 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:54:58,118 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:54:58,118 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:54:58,118 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:54:58,118 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:54:58,118 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:54:58,118 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. 2023-04-16 00:54:58,118 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. started... 2023-04-16 00:54:58,118 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 5dcf6997c5a081edf1d95a6a373c6b08 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-16 00:54:58,118 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5dcf6997c5a081edf1d95a6a373c6b08, store=A 2023-04-16 00:54:58,118 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:54:58,119 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5dcf6997c5a081edf1d95a6a373c6b08, store=B 2023-04-16 00:54:58,119 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:54:58,119 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5dcf6997c5a081edf1d95a6a373c6b08, store=C 2023-04-16 00:54:58,119 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:54:58,128 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=299 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/.tmp/A/92b52e941fd741bf99e8e0603a09364b 2023-04-16 00:54:58,139 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=299 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/.tmp/B/409cf632708e4f4abd31f556b844e326 2023-04-16 00:54:58,173 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(9158): Flush requested on 5dcf6997c5a081edf1d95a6a373c6b08 2023-04-16 00:54:58,173 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. as already flushing 2023-04-16 00:54:58,219 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:58,220 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 240 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58292 deadline: 1681606558217, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:58,322 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:58,322 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 242 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58292 deadline: 1681606558320, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:58,524 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:58,524 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 244 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58292 deadline: 1681606558523, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:58,549 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=299 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/.tmp/C/496709f960b244d79395d4a01ba31a9e 2023-04-16 00:54:58,553 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/.tmp/A/92b52e941fd741bf99e8e0603a09364b as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/A/92b52e941fd741bf99e8e0603a09364b 2023-04-16 00:54:58,556 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/A/92b52e941fd741bf99e8e0603a09364b, entries=150, sequenceid=299, filesize=11.9 K 2023-04-16 00:54:58,557 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/.tmp/B/409cf632708e4f4abd31f556b844e326 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/B/409cf632708e4f4abd31f556b844e326 2023-04-16 00:54:58,560 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/B/409cf632708e4f4abd31f556b844e326, entries=150, sequenceid=299, filesize=11.9 K 2023-04-16 00:54:58,561 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/.tmp/C/496709f960b244d79395d4a01ba31a9e as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/C/496709f960b244d79395d4a01ba31a9e 2023-04-16 00:54:58,564 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/C/496709f960b244d79395d4a01ba31a9e, entries=150, sequenceid=299, filesize=11.9 K 2023-04-16 00:54:58,565 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 5dcf6997c5a081edf1d95a6a373c6b08 in 447ms, sequenceid=299, compaction requested=true 2023-04-16 00:54:58,565 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 5dcf6997c5a081edf1d95a6a373c6b08: 2023-04-16 00:54:58,565 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. 2023-04-16 00:54:58,565 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:54:58,565 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:54:58,565 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:58,565 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:54:58,565 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:54:58,567 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:58,567 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:58,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:58,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:58,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:58,568 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:58,568 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:54:58,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:58,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:58,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:58,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:58,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:58,569 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:54:58,570 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@c21b1a8[Count = 0] remaining members to acquire global barrier 2023-04-16 00:54:58,570 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:54:58,570 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:58,571 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:58,571 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:58,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:58,571 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:54:58,571 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:54:58,571 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:54:58,571 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:58,571 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:54:58,572 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:54:58,572 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:54:58,572 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:58,572 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:54:58,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:58,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:58,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:58,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:58,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:58,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:58,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:58,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:58,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:58,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:58,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:54:58,575 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:54:58,575 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:54:58,575 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:54:58,575 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:54:58,575 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:54:58,575 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:54:58,576 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:58,576 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:58,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:58,576 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:58,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:58,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:58,576 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:54:58,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:58,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:58,577 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:54:58,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:54:58,577 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:58,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:58,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:58,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:58,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:58,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:58,578 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:58,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:58,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:58,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:58,584 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:58,584 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:58,584 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:54:58,584 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:54:58,584 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:58,584 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:54:58,584 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:54:58,584 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:58,584 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:54:58,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:54:58,584 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:54:58,584 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:58,584 DEBUG [Thread-1736] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:54:58,584 DEBUG [Thread-1736] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:54:58,584 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:58,585 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:58,585 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:54:58,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:54:58,685 DEBUG [Thread-1736] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:54:58,685 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:54:58,687 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:54:58,687 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:54:58,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:54:58,688 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:54:58,688 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:54:58,688 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:54:58,688 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:54:58,688 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:58,688 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:58,689 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:58,689 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:54:58,689 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:54:58,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:54:58,689 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:58,690 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:54:58,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:58,690 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:58,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:54:58,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:58,691 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:54:58,691 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:54:58,691 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:54:58,691 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:54:58,691 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:54:58,691 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:54:58,691 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:54:58,691 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. 2023-04-16 00:54:58,691 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. started... 2023-04-16 00:54:58,691 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 5dcf6997c5a081edf1d95a6a373c6b08 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-16 00:54:58,692 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5dcf6997c5a081edf1d95a6a373c6b08, store=A 2023-04-16 00:54:58,692 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:54:58,692 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5dcf6997c5a081edf1d95a6a373c6b08, store=B 2023-04-16 00:54:58,692 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:54:58,692 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5dcf6997c5a081edf1d95a6a373c6b08, store=C 2023-04-16 00:54:58,692 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:54:58,699 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=319 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/.tmp/A/f8f606d88e564c66ace684989cfa2ce1 2023-04-16 00:54:58,828 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(9158): Flush requested on 5dcf6997c5a081edf1d95a6a373c6b08 2023-04-16 00:54:58,828 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. as already flushing 2023-04-16 00:54:58,850 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:58,851 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 255 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58292 deadline: 1681606558849, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:58,953 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:58,953 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 257 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58292 deadline: 1681606558951, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:59,110 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=319 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/.tmp/B/c467234f3865490e819ebb8a275995e7 2023-04-16 00:54:59,155 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:59,155 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 259 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58292 deadline: 1681606559154, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:59,437 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:59,437 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58282 deadline: 1681606559435, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:59,439 DEBUG [Thread-1726] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8146 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08., hostname=jenkins-hbase4.apache.org,34975,1681606419217, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor32.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-16 00:54:59,440 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:59,441 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58336 deadline: 1681606559440, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:59,442 DEBUG [Thread-1732] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8150 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08., hostname=jenkins-hbase4.apache.org,34975,1681606419217, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor32.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-16 00:54:59,445 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:59,445 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58306 deadline: 1681606559443, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:59,447 DEBUG [Thread-1728] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8158 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08., hostname=jenkins-hbase4.apache.org,34975,1681606419217, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor32.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-16 00:54:59,458 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:59,458 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 261 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58292 deadline: 1681606559456, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:59,470 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:54:59,470 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58322 deadline: 1681606559468, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:59,471 DEBUG [Thread-1734] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8182 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08., hostname=jenkins-hbase4.apache.org,34975,1681606419217, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor32.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-16 00:54:59,521 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=319 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/.tmp/C/6b37490beafd4793a113780f46c9b8fe 2023-04-16 00:54:59,525 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/.tmp/A/f8f606d88e564c66ace684989cfa2ce1 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/A/f8f606d88e564c66ace684989cfa2ce1 2023-04-16 00:54:59,528 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/A/f8f606d88e564c66ace684989cfa2ce1, entries=150, sequenceid=319, filesize=11.9 K 2023-04-16 00:54:59,529 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/.tmp/B/c467234f3865490e819ebb8a275995e7 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/B/c467234f3865490e819ebb8a275995e7 2023-04-16 00:54:59,532 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/B/c467234f3865490e819ebb8a275995e7, entries=150, sequenceid=319, filesize=11.9 K 2023-04-16 00:54:59,536 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/.tmp/C/6b37490beafd4793a113780f46c9b8fe as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/C/6b37490beafd4793a113780f46c9b8fe 2023-04-16 00:54:59,539 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/C/6b37490beafd4793a113780f46c9b8fe, entries=150, sequenceid=319, filesize=11.9 K 2023-04-16 00:54:59,540 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for 5dcf6997c5a081edf1d95a6a373c6b08 in 849ms, sequenceid=319, compaction requested=true 2023-04-16 00:54:59,540 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 5dcf6997c5a081edf1d95a6a373c6b08: 2023-04-16 00:54:59,540 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. 2023-04-16 00:54:59,540 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:54:59,540 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:54:59,540 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:59,540 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:54:59,540 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:54:59,542 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:59,542 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:59,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:59,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:59,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:59,542 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:59,542 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:54:59,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:59,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:59,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:59,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:59,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:59,544 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:54:59,544 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@31c58a57[Count = 0] remaining members to acquire global barrier 2023-04-16 00:54:59,544 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:54:59,544 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:59,545 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:59,545 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:59,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:59,545 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:54:59,545 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:54:59,545 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:54:59,545 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:59,546 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:54:59,547 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:54:59,547 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:59,547 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:54:59,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:59,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:59,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:59,547 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:54:59,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:59,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:59,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:59,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:59,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:59,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:59,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:59,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:54:59,550 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:54:59,550 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:54:59,550 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:54:59,550 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:54:59,550 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:54:59,550 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:54:59,553 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:59,553 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:59,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:59,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:54:59,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:54:59,553 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:59,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:59,553 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:54:59,553 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:54:59,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:54:59,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:54:59,553 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:59,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:59,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:59,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:54:59,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:59,555 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:59,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:59,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:54:59,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:54:59,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:59,557 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:59,557 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:59,557 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:59,557 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:59,557 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:59,557 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:54:59,557 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:54:59,557 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:54:59,557 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:54:59,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:54:59,558 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:59,558 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:54:59,558 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:54:59,558 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:54:59,558 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:54:59,559 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:54:59,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:54:59,559 DEBUG [Thread-1736] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:54:59,559 DEBUG [Thread-1736] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:54:59,659 DEBUG [Thread-1736] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:54:59,659 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:54:59,661 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:54:59,662 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:54:59,662 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:54:59,662 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:54:59,662 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:54:59,662 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:54:59,662 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:54:59,663 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:59,663 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:59,664 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:54:59,664 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:59,664 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:54:59,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:54:59,664 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:54:59,664 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:54:59,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:59,665 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:54:59,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:54:59,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:54:59,665 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:54:59,665 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:54:59,665 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:54:59,665 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:54:59,665 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:54:59,665 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:54:59,666 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:54:59,666 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:54:59,666 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. 2023-04-16 00:54:59,666 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. started... 2023-04-16 00:54:59,666 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 5dcf6997c5a081edf1d95a6a373c6b08 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-16 00:54:59,666 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5dcf6997c5a081edf1d95a6a373c6b08, store=A 2023-04-16 00:54:59,666 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:54:59,666 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5dcf6997c5a081edf1d95a6a373c6b08, store=B 2023-04-16 00:54:59,666 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:54:59,666 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5dcf6997c5a081edf1d95a6a373c6b08, store=C 2023-04-16 00:54:59,666 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:54:59,961 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(9158): Flush requested on 5dcf6997c5a081edf1d95a6a373c6b08 2023-04-16 00:54:59,961 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. as already flushing 2023-04-16 00:55:00,019 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:00,019 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 280 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58292 deadline: 1681606560017, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:00,074 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=331 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/.tmp/A/bfa1e1144893410a9611e4123990091a 2023-04-16 00:55:00,121 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:00,122 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 282 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58292 deadline: 1681606560120, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:00,326 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:00,326 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 284 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58292 deadline: 1681606560322, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:00,485 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=331 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/.tmp/B/1f91bba7117f441797d27bd90fe7b530 2023-04-16 00:55:00,629 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:00,630 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 286 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58292 deadline: 1681606560629, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:00,894 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=331 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/.tmp/C/4901da3437ca4446b7c7bcce0d7136d5 2023-04-16 00:55:00,898 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/.tmp/A/bfa1e1144893410a9611e4123990091a as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/A/bfa1e1144893410a9611e4123990091a 2023-04-16 00:55:00,901 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/A/bfa1e1144893410a9611e4123990091a, entries=150, sequenceid=331, filesize=11.9 K 2023-04-16 00:55:00,902 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/.tmp/B/1f91bba7117f441797d27bd90fe7b530 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/B/1f91bba7117f441797d27bd90fe7b530 2023-04-16 00:55:00,905 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/B/1f91bba7117f441797d27bd90fe7b530, entries=150, sequenceid=331, filesize=11.9 K 2023-04-16 00:55:00,906 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/.tmp/C/4901da3437ca4446b7c7bcce0d7136d5 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/C/4901da3437ca4446b7c7bcce0d7136d5 2023-04-16 00:55:00,909 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/C/4901da3437ca4446b7c7bcce0d7136d5, entries=150, sequenceid=331, filesize=11.9 K 2023-04-16 00:55:00,910 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 5dcf6997c5a081edf1d95a6a373c6b08 in 1244ms, sequenceid=331, compaction requested=true 2023-04-16 00:55:00,910 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 5dcf6997c5a081edf1d95a6a373c6b08: 2023-04-16 00:55:00,910 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. 2023-04-16 00:55:00,910 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:55:00,910 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:55:00,910 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:00,910 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:55:00,910 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:55:00,912 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:00,912 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:00,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:00,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:00,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:00,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:00,912 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:00,912 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:55:00,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:00,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:00,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:00,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:00,914 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:55:00,914 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1e08778a[Count = 0] remaining members to acquire global barrier 2023-04-16 00:55:00,914 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:55:00,914 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:00,916 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:00,916 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:00,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:00,916 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:55:00,916 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:55:00,916 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:55:00,916 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:00,917 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:55:00,918 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:00,918 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:55:00,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:00,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:00,918 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:00,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:00,918 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:55:00,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:00,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:00,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:00,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:00,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:00,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:00,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:00,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:55:00,920 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:55:00,920 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:55:00,920 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:55:00,920 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:55:00,920 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:55:00,920 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:55:00,922 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:00,922 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:00,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:00,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:00,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:00,922 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:00,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:00,922 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:00,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:00,922 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:00,922 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:00,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:00,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:00,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:00,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:00,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:00,924 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:00,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:00,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:00,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:00,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:00,926 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:00,926 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:00,926 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:00,926 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:00,926 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:00,926 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:00,926 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:00,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:00,926 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:00,926 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:55:00,926 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:55:00,926 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:00,926 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:00,926 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:00,926 DEBUG [Thread-1736] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:55:00,926 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:00,926 DEBUG [Thread-1736] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:55:00,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:00,926 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:55:01,026 DEBUG [Thread-1736] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:55:01,027 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:55:01,028 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:55:01,030 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:55:01,030 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:55:01,030 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:55:01,030 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:01,030 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:55:01,030 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:55:01,030 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:01,030 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:01,032 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:01,032 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:01,032 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:01,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:01,032 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:01,033 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:55:01,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:01,033 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:01,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:55:01,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:01,033 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:55:01,033 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:55:01,033 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:55:01,034 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:55:01,034 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:01,034 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:55:01,034 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:55:01,034 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:55:01,034 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. 2023-04-16 00:55:01,034 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. started... 2023-04-16 00:55:01,034 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 5dcf6997c5a081edf1d95a6a373c6b08 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-16 00:55:01,034 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5dcf6997c5a081edf1d95a6a373c6b08, store=A 2023-04-16 00:55:01,034 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:55:01,034 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5dcf6997c5a081edf1d95a6a373c6b08, store=B 2023-04-16 00:55:01,034 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:55:01,035 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5dcf6997c5a081edf1d95a6a373c6b08, store=C 2023-04-16 00:55:01,035 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:55:01,136 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(9158): Flush requested on 5dcf6997c5a081edf1d95a6a373c6b08 2023-04-16 00:55:01,136 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. as already flushing 2023-04-16 00:55:01,171 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:01,171 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 297 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58292 deadline: 1681606561167, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:01,273 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:01,273 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 299 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58292 deadline: 1681606561272, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:01,442 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=351 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/.tmp/A/8d979d19aa2640b4b8d71ecf55b7c38f 2023-04-16 00:55:01,452 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=351 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/.tmp/B/601f847e752342c7859b48241d444303 2023-04-16 00:55:01,477 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:01,477 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 301 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58292 deadline: 1681606561475, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:01,782 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:01,782 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 303 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58292 deadline: 1681606561780, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:01,862 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=351 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/.tmp/C/5c858daf872b4aa0bad320f785a9a9aa 2023-04-16 00:55:01,866 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/.tmp/A/8d979d19aa2640b4b8d71ecf55b7c38f as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/A/8d979d19aa2640b4b8d71ecf55b7c38f 2023-04-16 00:55:01,869 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/A/8d979d19aa2640b4b8d71ecf55b7c38f, entries=150, sequenceid=351, filesize=11.9 K 2023-04-16 00:55:01,869 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/.tmp/B/601f847e752342c7859b48241d444303 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/B/601f847e752342c7859b48241d444303 2023-04-16 00:55:01,872 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/B/601f847e752342c7859b48241d444303, entries=150, sequenceid=351, filesize=11.9 K 2023-04-16 00:55:01,873 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/.tmp/C/5c858daf872b4aa0bad320f785a9a9aa as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/C/5c858daf872b4aa0bad320f785a9a9aa 2023-04-16 00:55:01,876 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/C/5c858daf872b4aa0bad320f785a9a9aa, entries=150, sequenceid=351, filesize=11.9 K 2023-04-16 00:55:01,877 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for 5dcf6997c5a081edf1d95a6a373c6b08 in 843ms, sequenceid=351, compaction requested=true 2023-04-16 00:55:01,877 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 5dcf6997c5a081edf1d95a6a373c6b08: 2023-04-16 00:55:01,877 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. 2023-04-16 00:55:01,877 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:55:01,877 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:55:01,877 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:01,877 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:55:01,877 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:55:01,878 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:01,878 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:01,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:01,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:01,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:01,887 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:01,887 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:55:01,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:01,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:01,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:01,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:01,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:01,888 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:55:01,889 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@32ede317[Count = 0] remaining members to acquire global barrier 2023-04-16 00:55:01,889 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:55:01,889 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:01,890 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:01,890 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:01,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:01,890 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:55:01,890 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:55:01,890 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:55:01,890 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:01,890 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:55:01,891 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:01,891 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:55:01,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:01,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:01,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:01,891 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:01,891 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:55:01,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:01,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:01,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:01,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:01,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:01,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:01,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:01,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:55:01,894 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:55:01,894 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:55:01,894 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:55:01,894 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:55:01,894 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:55:01,894 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:55:01,897 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:01,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:01,897 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:01,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:01,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:01,898 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:01,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:01,897 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:01,898 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:01,898 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:01,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:01,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:01,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:01,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:01,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:01,899 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:01,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:01,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:01,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:01,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:01,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:01,901 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:01,902 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:01,902 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:01,902 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:01,902 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:01,902 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:01,902 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:01,902 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:01,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:01,902 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:55:01,902 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:55:01,902 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:01,902 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:01,902 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:01,902 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:55:01,902 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:01,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:01,902 DEBUG [Thread-1736] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:55:01,902 DEBUG [Thread-1736] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:55:02,003 DEBUG [Thread-1736] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:55:02,003 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:55:02,005 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:55:02,006 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:55:02,006 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:55:02,006 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:55:02,006 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:02,006 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:55:02,006 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:55:02,007 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:02,007 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:02,011 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:02,011 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:02,011 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:02,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:02,011 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:02,011 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:55:02,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:02,011 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:02,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:55:02,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:02,012 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:55:02,012 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:55:02,012 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:55:02,012 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:55:02,012 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:02,012 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:55:02,012 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:55:02,012 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:55:02,012 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. 2023-04-16 00:55:02,013 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. started... 2023-04-16 00:55:02,013 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 5dcf6997c5a081edf1d95a6a373c6b08 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-16 00:55:02,013 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5dcf6997c5a081edf1d95a6a373c6b08, store=A 2023-04-16 00:55:02,013 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:55:02,013 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5dcf6997c5a081edf1d95a6a373c6b08, store=B 2023-04-16 00:55:02,013 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:55:02,013 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5dcf6997c5a081edf1d95a6a373c6b08, store=C 2023-04-16 00:55:02,013 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:55:02,287 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(9158): Flush requested on 5dcf6997c5a081edf1d95a6a373c6b08 2023-04-16 00:55:02,287 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. as already flushing 2023-04-16 00:55:02,357 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:02,357 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 322 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58292 deadline: 1681606562354, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:02,421 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=363 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/.tmp/A/695a75e780414782951c6fd010c6cea6 2023-04-16 00:55:02,430 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=363 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/.tmp/B/1cd1e91e9e6b4fa38b9e9a0ac3b2bc94 2023-04-16 00:55:02,459 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:02,460 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 324 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58292 deadline: 1681606562458, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:02,663 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:02,663 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 326 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58292 deadline: 1681606562661, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:02,840 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=363 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/.tmp/C/4ff8907dd9bb49a29564c119a9b3421e 2023-04-16 00:55:02,844 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/.tmp/A/695a75e780414782951c6fd010c6cea6 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/A/695a75e780414782951c6fd010c6cea6 2023-04-16 00:55:02,847 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/A/695a75e780414782951c6fd010c6cea6, entries=150, sequenceid=363, filesize=11.9 K 2023-04-16 00:55:02,848 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/.tmp/B/1cd1e91e9e6b4fa38b9e9a0ac3b2bc94 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/B/1cd1e91e9e6b4fa38b9e9a0ac3b2bc94 2023-04-16 00:55:02,851 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/B/1cd1e91e9e6b4fa38b9e9a0ac3b2bc94, entries=150, sequenceid=363, filesize=11.9 K 2023-04-16 00:55:02,852 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/.tmp/C/4ff8907dd9bb49a29564c119a9b3421e as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/C/4ff8907dd9bb49a29564c119a9b3421e 2023-04-16 00:55:02,854 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/C/4ff8907dd9bb49a29564c119a9b3421e, entries=150, sequenceid=363, filesize=11.9 K 2023-04-16 00:55:02,855 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 5dcf6997c5a081edf1d95a6a373c6b08 in 842ms, sequenceid=363, compaction requested=true 2023-04-16 00:55:02,855 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 5dcf6997c5a081edf1d95a6a373c6b08: 2023-04-16 00:55:02,855 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. 2023-04-16 00:55:02,855 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:55:02,855 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:55:02,855 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:02,855 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:55:02,855 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:55:02,857 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:02,857 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:02,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:02,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:02,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:02,857 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:02,857 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:55:02,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:02,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:02,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:02,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:02,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:02,859 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:55:02,859 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3340a813[Count = 0] remaining members to acquire global barrier 2023-04-16 00:55:02,859 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:55:02,859 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:02,860 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:02,860 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:02,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:02,860 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:55:02,860 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:55:02,860 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:55:02,860 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:02,860 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:55:02,862 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:55:02,862 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:02,862 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:02,862 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:55:02,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:02,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:02,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:02,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:02,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:02,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:02,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:02,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:02,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:02,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:02,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:55:02,865 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:55:02,865 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:55:02,865 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:55:02,865 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:55:02,865 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:55:02,865 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:55:02,866 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:02,866 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:02,866 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:02,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:02,866 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:02,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:02,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:02,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:02,867 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:02,867 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:02,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:02,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:02,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:02,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:02,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:02,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:02,868 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:02,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:02,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:02,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:02,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:02,870 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:02,870 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:02,870 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:02,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:02,870 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:02,870 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:02,870 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:55:02,870 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:55:02,870 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:02,870 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:02,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:02,870 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:55:02,870 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:02,871 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:02,871 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:02,871 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:02,871 DEBUG [Thread-1736] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:55:02,871 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:02,871 DEBUG [Thread-1736] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:55:02,967 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(9158): Flush requested on 5dcf6997c5a081edf1d95a6a373c6b08 2023-04-16 00:55:02,967 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 5dcf6997c5a081edf1d95a6a373c6b08 3/3 column families, dataSize=120.76 KB heapSize=360.05 KB 2023-04-16 00:55:02,967 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5dcf6997c5a081edf1d95a6a373c6b08, store=A 2023-04-16 00:55:02,968 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 00:55:02,968 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5dcf6997c5a081edf1d95a6a373c6b08, store=B 2023-04-16 00:55:02,968 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 00:55:02,968 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5dcf6997c5a081edf1d95a6a373c6b08, store=C 2023-04-16 00:55:02,968 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 00:55:02,971 DEBUG [Thread-1736] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:55:02,971 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:55:02,973 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:55:02,974 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:55:02,975 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:55:02,975 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:55:02,975 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:02,975 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:55:02,975 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:55:02,975 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=384 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/.tmp/A/d59b4c0a106c4062b8ac41f6b4f0f578 2023-04-16 00:55:02,977 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:02,977 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:02,978 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:02,978 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:02,979 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:02,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:02,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:02,979 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:02,980 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:02,980 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:55:02,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:55:02,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:02,980 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:55:02,980 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:55:02,980 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:55:02,980 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:55:02,980 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:02,980 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:55:02,981 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:55:02,981 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:55:02,981 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. 2023-04-16 00:55:02,981 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. started... 2023-04-16 00:55:02,981 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. as already flushing 2023-04-16 00:55:02,981 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. 2023-04-16 00:55:02,981 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:55:02,981 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:55:02,981 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:02,981 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:55:02,981 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:55:02,984 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:02,984 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:02,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:02,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:02,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:02,984 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:02,984 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:55:02,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:02,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:02,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:02,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:02,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:02,986 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:55:02,986 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2eca1b1e[Count = 0] remaining members to acquire global barrier 2023-04-16 00:55:02,986 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:55:02,986 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:02,987 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:02,987 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:02,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:02,987 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:55:02,987 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:02,988 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:55:02,987 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:55:02,988 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:55:02,989 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:02,989 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:55:02,989 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:02,989 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:55:02,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:02,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:02,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:02,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:02,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:02,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:02,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:02,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:02,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:02,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:02,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:55:02,991 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:55:02,991 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:55:02,991 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:55:02,991 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:55:02,991 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:55:02,991 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:55:02,993 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:02,993 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:02,993 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:02,993 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:02,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:02,993 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:02,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:02,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:02,994 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:02,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:02,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:02,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:02,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:02,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:02,995 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:02,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:02,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:02,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:02,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:02,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:02,998 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:02,998 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:02,998 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:02,998 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:55:02,998 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:55:02,998 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:02,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:02,998 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:02,998 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:02,998 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:02,998 DEBUG [Thread-1736] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:55:02,998 DEBUG [Thread-1736] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:55:02,998 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:55:02,998 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:02,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:02,998 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:02,998 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:02,998 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:02,998 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:03,004 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:03,004 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 337 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58292 deadline: 1681606563002, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:03,098 DEBUG [Thread-1736] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:55:03,099 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:55:03,100 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:55:03,101 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:55:03,101 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:55:03,101 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:55:03,101 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:03,101 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:55:03,101 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:55:03,102 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:03,102 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:03,103 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:03,103 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:03,103 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:03,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:03,103 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:03,103 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:55:03,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:03,104 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:03,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:55:03,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:03,104 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:55:03,104 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:55:03,104 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:55:03,104 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:55:03,105 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:03,105 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:55:03,105 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:55:03,105 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:55:03,105 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. 2023-04-16 00:55:03,105 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. started... 2023-04-16 00:55:03,105 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. as already flushing 2023-04-16 00:55:03,105 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. 2023-04-16 00:55:03,105 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:55:03,105 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:55:03,105 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:03,105 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:55:03,105 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:55:03,106 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:03,106 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:03,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:03,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:03,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:03,107 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:03,107 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:55:03,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:03,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:03,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:03,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:03,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:03,108 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:55:03,108 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@73a39d6a[Count = 0] remaining members to acquire global barrier 2023-04-16 00:55:03,108 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:55:03,108 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:03,109 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:03,109 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 339 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58292 deadline: 1681606563105, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:03,109 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:03,109 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:03,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:03,109 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:55:03,110 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:03,110 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:55:03,110 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:55:03,110 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:55:03,112 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:03,112 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:55:03,112 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:03,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:03,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:03,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:03,112 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:55:03,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:03,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:03,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:03,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:03,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:03,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:03,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:03,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:55:03,114 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:55:03,114 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:55:03,114 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:55:03,114 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:55:03,114 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:55:03,114 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:55:03,116 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:03,116 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:03,116 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:03,116 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:03,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:03,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:03,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:03,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:03,116 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:03,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:03,116 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:03,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:03,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:03,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:03,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:03,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:03,117 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:03,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:03,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:03,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:03,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:03,120 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:03,120 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:03,120 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:03,120 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:03,120 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:03,120 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:55:03,120 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:55:03,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:03,120 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:03,120 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:03,120 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:03,120 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:03,120 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:03,120 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:03,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:03,120 DEBUG [Thread-1736] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:55:03,120 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:03,120 DEBUG [Thread-1736] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:55:03,221 DEBUG [Thread-1736] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:55:03,221 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:55:03,222 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:55:03,224 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:55:03,224 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:55:03,224 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:55:03,224 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:03,224 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:55:03,224 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:55:03,224 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:03,224 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:03,226 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:03,226 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:03,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:03,226 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:03,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:03,226 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:03,226 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:55:03,226 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:03,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:55:03,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:03,227 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:55:03,227 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:55:03,227 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:55:03,227 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:55:03,227 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:03,227 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:55:03,227 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:55:03,227 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:55:03,228 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. 2023-04-16 00:55:03,228 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. started... 2023-04-16 00:55:03,228 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. as already flushing 2023-04-16 00:55:03,228 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. 2023-04-16 00:55:03,228 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:55:03,228 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:55:03,228 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:03,228 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:55:03,228 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:55:03,233 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:03,233 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:03,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:03,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:03,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:03,233 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:03,233 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:55:03,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:03,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:03,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:03,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:03,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:03,235 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:55:03,235 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2e8b3d52[Count = 0] remaining members to acquire global barrier 2023-04-16 00:55:03,235 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:55:03,235 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:03,236 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:03,236 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:03,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:03,236 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:55:03,236 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:55:03,236 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:55:03,237 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:03,237 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:55:03,238 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:03,238 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:55:03,238 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:03,238 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:55:03,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:03,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:03,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:03,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:03,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:03,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:03,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:03,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:03,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:03,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:03,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:55:03,240 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:55:03,240 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:55:03,240 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:55:03,240 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:55:03,240 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:55:03,240 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:55:03,242 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:03,242 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:03,242 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:03,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:03,242 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:03,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:03,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:03,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:03,242 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:03,242 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:03,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:03,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:03,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:03,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:03,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:03,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:03,243 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:03,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:03,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:03,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:03,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:03,247 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:03,247 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:03,247 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:03,247 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:03,247 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:03,247 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:03,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:03,247 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:55:03,247 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:55:03,247 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:03,247 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:55:03,247 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:03,247 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:03,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:03,247 DEBUG [Thread-1736] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:55:03,247 DEBUG [Thread-1736] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:55:03,247 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:03,248 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:03,248 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:03,311 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:03,312 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 341 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58292 deadline: 1681606563310, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:03,348 DEBUG [Thread-1736] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:55:03,348 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:55:03,349 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:55:03,351 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:55:03,351 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:55:03,351 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:55:03,351 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:03,351 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:55:03,351 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:55:03,351 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:03,351 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:03,353 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:03,353 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:03,353 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:03,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:03,353 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:03,353 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:55:03,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:03,353 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:03,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:55:03,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:03,354 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:55:03,354 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:55:03,354 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:55:03,354 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:55:03,354 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:03,354 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:55:03,354 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:55:03,354 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:55:03,354 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. 2023-04-16 00:55:03,354 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. started... 2023-04-16 00:55:03,355 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. as already flushing 2023-04-16 00:55:03,355 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. 2023-04-16 00:55:03,355 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:55:03,355 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:55:03,355 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:03,355 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:55:03,355 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:55:03,356 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:03,356 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:03,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:03,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:03,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:03,356 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:03,356 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:55:03,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:03,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:03,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:03,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:03,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:03,358 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:55:03,358 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@183d18a0[Count = 0] remaining members to acquire global barrier 2023-04-16 00:55:03,358 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:55:03,358 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:03,359 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:03,359 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:03,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:03,359 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:55:03,359 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:55:03,359 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:55:03,359 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:03,359 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:55:03,361 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:03,361 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:55:03,361 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:03,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:03,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:03,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:03,361 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:55:03,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:03,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:03,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:03,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:03,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:03,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:03,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:03,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:55:03,363 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:55:03,363 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:55:03,363 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:55:03,364 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:55:03,364 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:55:03,364 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:55:03,365 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:03,365 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:03,365 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:03,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:03,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:03,365 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:03,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:03,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:03,365 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:03,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:03,366 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:03,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:03,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:03,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:03,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:03,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:03,367 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:03,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:03,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:03,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:03,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:03,369 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:03,369 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:03,369 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:03,369 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:03,369 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:03,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:03,369 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:03,369 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:03,369 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:55:03,369 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:55:03,369 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:03,369 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:03,369 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:55:03,369 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:03,369 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:03,369 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:03,369 DEBUG [Thread-1736] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:55:03,369 DEBUG [Thread-1736] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:55:03,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:03,386 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=384 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/.tmp/B/5b9326a81a6a4ac086028ac1e417f543 2023-04-16 00:55:03,469 DEBUG [Thread-1736] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:55:03,470 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:55:03,471 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:55:03,472 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:55:03,472 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:55:03,472 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:55:03,473 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:03,473 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:55:03,473 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:55:03,473 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:03,473 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:03,474 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:03,474 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:03,474 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:03,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:03,474 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:03,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:03,474 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:55:03,475 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:03,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:55:03,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:03,475 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:55:03,475 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:55:03,475 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:55:03,475 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:55:03,475 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:03,476 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:55:03,476 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:55:03,476 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:55:03,476 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. 2023-04-16 00:55:03,476 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. started... 2023-04-16 00:55:03,476 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. as already flushing 2023-04-16 00:55:03,476 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. 2023-04-16 00:55:03,476 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:55:03,476 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:55:03,476 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:03,476 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:55:03,476 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:55:03,479 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:03,479 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:03,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:03,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:03,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:03,479 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:03,479 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:55:03,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:03,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:03,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:03,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:03,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:03,480 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:55:03,480 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:55:03,480 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@66d1e252[Count = 0] remaining members to acquire global barrier 2023-04-16 00:55:03,480 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:03,481 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:03,481 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:03,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:03,482 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:55:03,482 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:55:03,482 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:03,482 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:55:03,482 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:55:03,483 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:03,483 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:55:03,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:03,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:03,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:03,483 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:03,483 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:55:03,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:03,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:03,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:03,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:03,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:03,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:03,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:03,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:55:03,486 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:55:03,486 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:55:03,486 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:55:03,486 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:55:03,486 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:55:03,486 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:55:03,487 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:03,487 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:03,487 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:03,487 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:03,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:03,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:03,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:03,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:03,488 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:03,488 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:03,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:03,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:03,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:03,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:03,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:03,489 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:03,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:03,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:03,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:03,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:03,491 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:03,491 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:03,491 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:03,491 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:03,491 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:03,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:03,491 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:03,492 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:03,492 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:03,492 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:03,492 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:03,492 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:03,492 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:55:03,492 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:03,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:03,492 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:55:03,492 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:55:03,493 DEBUG [Thread-1736] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:55:03,493 DEBUG [Thread-1736] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:55:03,593 DEBUG [Thread-1736] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:55:03,593 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:55:03,595 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:55:03,596 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:55:03,596 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:55:03,596 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:55:03,596 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:03,596 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:55:03,596 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:55:03,597 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:03,597 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:03,598 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:03,598 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:03,598 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:03,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:03,598 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:03,598 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:55:03,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:03,598 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:03,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:55:03,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:03,599 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:55:03,599 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:55:03,599 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:55:03,599 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:55:03,599 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:03,599 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:55:03,599 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:55:03,599 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:55:03,600 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. 2023-04-16 00:55:03,600 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. started... 2023-04-16 00:55:03,600 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. as already flushing 2023-04-16 00:55:03,600 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. 2023-04-16 00:55:03,600 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:55:03,600 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:55:03,600 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:03,600 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:55:03,600 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:55:03,601 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:03,601 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:03,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:03,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:03,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:03,602 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:03,602 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:55:03,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:03,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:03,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:03,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:03,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:03,603 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:55:03,603 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:55:03,603 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@9f67b2f[Count = 0] remaining members to acquire global barrier 2023-04-16 00:55:03,603 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:03,605 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:03,605 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:03,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:03,605 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:55:03,605 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:55:03,605 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:03,605 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:55:03,605 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:55:03,613 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:03,613 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:55:03,613 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:03,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:03,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:03,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:03,613 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:55:03,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:03,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:03,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:03,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:03,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:03,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:03,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:03,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:55:03,615 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:55:03,615 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:55:03,615 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:55:03,615 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:55:03,615 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:55:03,615 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:55:03,616 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:03,617 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 343 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58292 deadline: 1681606563614, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:03,617 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:03,617 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:03,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:03,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:03,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:03,617 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:03,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:03,617 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:03,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:03,618 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:03,618 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:03,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:03,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:03,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:03,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:03,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:03,619 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:03,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:03,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:03,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:03,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:03,622 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:03,622 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:03,622 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:03,622 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:55:03,622 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:55:03,622 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:03,622 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:03,622 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:03,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:03,622 DEBUG [Thread-1736] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:55:03,622 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:03,622 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:03,622 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:55:03,622 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:03,622 DEBUG [Thread-1736] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:55:03,622 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:03,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:03,622 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:03,622 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:03,722 DEBUG [Thread-1736] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:55:03,723 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:55:03,724 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:55:03,725 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:55:03,726 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:55:03,726 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:55:03,726 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:03,726 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:55:03,726 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:55:03,726 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:03,726 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:03,727 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:03,727 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:03,727 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:03,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:03,727 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:03,727 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:55:03,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:03,728 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:03,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:55:03,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:03,728 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:55:03,728 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:55:03,729 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:55:03,729 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:55:03,729 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:03,729 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:55:03,729 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:55:03,729 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:55:03,729 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. 2023-04-16 00:55:03,729 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. started... 2023-04-16 00:55:03,729 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. as already flushing 2023-04-16 00:55:03,729 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. 2023-04-16 00:55:03,729 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:55:03,729 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:55:03,729 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:03,729 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:55:03,729 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:55:03,731 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:03,731 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:03,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:03,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:03,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:03,731 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:03,731 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:55:03,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:03,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:03,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:03,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:03,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:03,733 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:55:03,733 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:55:03,733 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7e60a6ee[Count = 0] remaining members to acquire global barrier 2023-04-16 00:55:03,733 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:03,734 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:03,734 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:03,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:03,734 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:03,734 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:55:03,734 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:55:03,734 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:55:03,734 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:55:03,735 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:03,735 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:55:03,735 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:03,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:03,735 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:55:03,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:03,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:03,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:03,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:03,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:03,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:03,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:03,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:03,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:03,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:55:03,738 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:55:03,738 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:55:03,738 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:55:03,738 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:55:03,738 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:55:03,738 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:55:03,739 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:03,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:03,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:03,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:03,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:03,740 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:03,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:03,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:03,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:03,741 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:03,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:03,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:03,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:03,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:03,744 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:03,744 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:03,744 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:03,744 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:03,745 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:03,745 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:03,745 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:03,745 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:55:03,745 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:55:03,745 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:03,745 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:55:03,745 DEBUG [Thread-1736] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:55:03,745 DEBUG [Thread-1736] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:55:03,747 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:03,747 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:03,747 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:03,747 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:03,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:03,747 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:55:03,747 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:03,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:03,748 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:03,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:03,795 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=384 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/.tmp/C/523c8eeb3ed0490696a815142fa0bc3a 2023-04-16 00:55:03,799 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/.tmp/A/d59b4c0a106c4062b8ac41f6b4f0f578 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/A/d59b4c0a106c4062b8ac41f6b4f0f578 2023-04-16 00:55:03,802 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/A/d59b4c0a106c4062b8ac41f6b4f0f578, entries=200, sequenceid=384, filesize=14.3 K 2023-04-16 00:55:03,803 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/.tmp/B/5b9326a81a6a4ac086028ac1e417f543 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/B/5b9326a81a6a4ac086028ac1e417f543 2023-04-16 00:55:03,806 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/B/5b9326a81a6a4ac086028ac1e417f543, entries=150, sequenceid=384, filesize=11.9 K 2023-04-16 00:55:03,807 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/.tmp/C/523c8eeb3ed0490696a815142fa0bc3a as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/C/523c8eeb3ed0490696a815142fa0bc3a 2023-04-16 00:55:03,810 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/C/523c8eeb3ed0490696a815142fa0bc3a, entries=150, sequenceid=384, filesize=11.9 K 2023-04-16 00:55:03,810 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~120.76 KB/123660, heapSize ~360 KB/368640, currentSize=53.67 KB/54960 for 5dcf6997c5a081edf1d95a6a373c6b08 in 843ms, sequenceid=384, compaction requested=true 2023-04-16 00:55:03,811 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 5dcf6997c5a081edf1d95a6a373c6b08: 2023-04-16 00:55:03,811 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 00:55:03,811 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 8 store files, 0 compacting, 8 eligible, 16 blocking 2023-04-16 00:55:03,811 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 8 store files, 0 compacting, 8 eligible, 16 blocking 2023-04-16 00:55:03,811 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 00:55:03,811 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 00:55:03,813 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 8 files of size 98102 starting at candidate #0 after considering 21 permutations with 21 in ratio 2023-04-16 00:55:03,813 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HStore(1912): 5dcf6997c5a081edf1d95a6a373c6b08/B is initiating minor compaction (all files) 2023-04-16 00:55:03,814 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 5dcf6997c5a081edf1d95a6a373c6b08/B in TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. 2023-04-16 00:55:03,814 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/B/b69c4c09edf74bb19ff8957d7ddfb6f3, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/B/11e14a78f35741d385ec8ab39d209a8e, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/B/409cf632708e4f4abd31f556b844e326, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/B/c467234f3865490e819ebb8a275995e7, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/B/1f91bba7117f441797d27bd90fe7b530, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/B/601f847e752342c7859b48241d444303, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/B/1cd1e91e9e6b4fa38b9e9a0ac3b2bc94, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/B/5b9326a81a6a4ac086028ac1e417f543] into tmpdir=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/.tmp, totalSize=95.8 K 2023-04-16 00:55:03,814 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 8 files of size 102982 starting at candidate #0 after considering 21 permutations with 21 in ratio 2023-04-16 00:55:03,814 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HStore(1912): 5dcf6997c5a081edf1d95a6a373c6b08/A is initiating minor compaction (all files) 2023-04-16 00:55:03,814 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 5dcf6997c5a081edf1d95a6a373c6b08/A in TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. 2023-04-16 00:55:03,814 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/A/ea063ba4012d48b7ac186c0b20fc87ea, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/A/3b6431bd48ec4e368d2fb70f1a3ee38d, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/A/92b52e941fd741bf99e8e0603a09364b, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/A/f8f606d88e564c66ace684989cfa2ce1, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/A/bfa1e1144893410a9611e4123990091a, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/A/8d979d19aa2640b4b8d71ecf55b7c38f, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/A/695a75e780414782951c6fd010c6cea6, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/A/d59b4c0a106c4062b8ac41f6b4f0f578] into tmpdir=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/.tmp, totalSize=100.6 K 2023-04-16 00:55:03,814 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting b69c4c09edf74bb19ff8957d7ddfb6f3, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=264, earliestPutTs=1681606496847 2023-04-16 00:55:03,814 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.Compactor(207): Compacting ea063ba4012d48b7ac186c0b20fc87ea, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=264, earliestPutTs=1681606496847 2023-04-16 00:55:03,815 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting 11e14a78f35741d385ec8ab39d209a8e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=284, earliestPutTs=1681606497491 2023-04-16 00:55:03,815 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.Compactor(207): Compacting 3b6431bd48ec4e368d2fb70f1a3ee38d, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=284, earliestPutTs=1681606497491 2023-04-16 00:55:03,815 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.Compactor(207): Compacting 92b52e941fd741bf99e8e0603a09364b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=299, earliestPutTs=1681606497551 2023-04-16 00:55:03,815 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting 409cf632708e4f4abd31f556b844e326, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=299, earliestPutTs=1681606497551 2023-04-16 00:55:03,816 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.Compactor(207): Compacting f8f606d88e564c66ace684989cfa2ce1, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=319, earliestPutTs=1681606498187 2023-04-16 00:55:03,816 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting c467234f3865490e819ebb8a275995e7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=319, earliestPutTs=1681606498187 2023-04-16 00:55:03,816 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.Compactor(207): Compacting bfa1e1144893410a9611e4123990091a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=331, earliestPutTs=1681606498840 2023-04-16 00:55:03,816 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting 1f91bba7117f441797d27bd90fe7b530, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=331, earliestPutTs=1681606498840 2023-04-16 00:55:03,817 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.Compactor(207): Compacting 8d979d19aa2640b4b8d71ecf55b7c38f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=351, earliestPutTs=1681606500002 2023-04-16 00:55:03,817 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting 601f847e752342c7859b48241d444303, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=351, earliestPutTs=1681606500002 2023-04-16 00:55:03,817 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.Compactor(207): Compacting 695a75e780414782951c6fd010c6cea6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=363, earliestPutTs=1681606501157 2023-04-16 00:55:03,817 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting 1cd1e91e9e6b4fa38b9e9a0ac3b2bc94, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=363, earliestPutTs=1681606501157 2023-04-16 00:55:03,817 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.Compactor(207): Compacting d59b4c0a106c4062b8ac41f6b4f0f578, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=384, earliestPutTs=1681606502346 2023-04-16 00:55:03,817 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting 5b9326a81a6a4ac086028ac1e417f543, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=384, earliestPutTs=1681606502350 2023-04-16 00:55:03,834 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] throttle.PressureAwareThroughputController(145): 5dcf6997c5a081edf1d95a6a373c6b08#B#compaction#295 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-16 00:55:03,835 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] throttle.PressureAwareThroughputController(145): 5dcf6997c5a081edf1d95a6a373c6b08#A#compaction#294 average throughput is 1.64 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-16 00:55:03,844 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/.tmp/A/ac924eb5b02647e3b3347a8c6e8dfd70 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/A/ac924eb5b02647e3b3347a8c6e8dfd70 2023-04-16 00:55:03,845 DEBUG [Thread-1736] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:55:03,846 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:55:03,847 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:55:03,848 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:55:03,849 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:55:03,849 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:55:03,849 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:03,849 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:55:03,849 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:55:03,849 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:03,849 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:03,850 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 8 (all) file(s) in 5dcf6997c5a081edf1d95a6a373c6b08/A of 5dcf6997c5a081edf1d95a6a373c6b08 into ac924eb5b02647e3b3347a8c6e8dfd70(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 00:55:03,850 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5dcf6997c5a081edf1d95a6a373c6b08: 2023-04-16 00:55:03,850 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08., storeName=5dcf6997c5a081edf1d95a6a373c6b08/A, priority=8, startTime=1681606503811; duration=0sec 2023-04-16 00:55:03,850 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 00:55:03,850 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 8 store files, 0 compacting, 8 eligible, 16 blocking 2023-04-16 00:55:03,850 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:03,850 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:03,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:03,850 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:03,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:03,851 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:03,851 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:55:03,851 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:03,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:55:03,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:03,852 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:55:03,852 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:55:03,852 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:55:03,852 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:55:03,852 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:03,852 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:55:03,852 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:55:03,852 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:55:03,852 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. 2023-04-16 00:55:03,852 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. started... 2023-04-16 00:55:03,852 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 5dcf6997c5a081edf1d95a6a373c6b08 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-16 00:55:03,853 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5dcf6997c5a081edf1d95a6a373c6b08, store=A 2023-04-16 00:55:03,853 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:55:03,852 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 8 files of size 98102 starting at candidate #0 after considering 21 permutations with 21 in ratio 2023-04-16 00:55:03,853 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5dcf6997c5a081edf1d95a6a373c6b08, store=B 2023-04-16 00:55:03,853 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:55:03,853 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HStore(1912): 5dcf6997c5a081edf1d95a6a373c6b08/C is initiating minor compaction (all files) 2023-04-16 00:55:03,853 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5dcf6997c5a081edf1d95a6a373c6b08, store=C 2023-04-16 00:55:03,853 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:55:03,853 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 5dcf6997c5a081edf1d95a6a373c6b08/C in TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. 2023-04-16 00:55:03,853 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/C/1fa18240d3e6447ba26dc86aa6049938, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/C/78c60dd18be44cb7acca58b59ac7efcd, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/C/496709f960b244d79395d4a01ba31a9e, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/C/6b37490beafd4793a113780f46c9b8fe, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/C/4901da3437ca4446b7c7bcce0d7136d5, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/C/5c858daf872b4aa0bad320f785a9a9aa, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/C/4ff8907dd9bb49a29564c119a9b3421e, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/C/523c8eeb3ed0490696a815142fa0bc3a] into tmpdir=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/.tmp, totalSize=95.8 K 2023-04-16 00:55:03,853 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.Compactor(207): Compacting 1fa18240d3e6447ba26dc86aa6049938, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=264, earliestPutTs=1681606496847 2023-04-16 00:55:03,854 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.Compactor(207): Compacting 78c60dd18be44cb7acca58b59ac7efcd, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=284, earliestPutTs=1681606497491 2023-04-16 00:55:03,855 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.Compactor(207): Compacting 496709f960b244d79395d4a01ba31a9e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=299, earliestPutTs=1681606497551 2023-04-16 00:55:03,855 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.Compactor(207): Compacting 6b37490beafd4793a113780f46c9b8fe, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=319, earliestPutTs=1681606498187 2023-04-16 00:55:03,855 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.Compactor(207): Compacting 4901da3437ca4446b7c7bcce0d7136d5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=331, earliestPutTs=1681606498840 2023-04-16 00:55:03,856 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.Compactor(207): Compacting 5c858daf872b4aa0bad320f785a9a9aa, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=351, earliestPutTs=1681606500002 2023-04-16 00:55:03,856 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.Compactor(207): Compacting 4ff8907dd9bb49a29564c119a9b3421e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=363, earliestPutTs=1681606501157 2023-04-16 00:55:03,856 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.Compactor(207): Compacting 523c8eeb3ed0490696a815142fa0bc3a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=384, earliestPutTs=1681606502350 2023-04-16 00:55:03,861 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=396 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/.tmp/A/5ad752265917432ca1117f1013767530 2023-04-16 00:55:03,880 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] throttle.PressureAwareThroughputController(145): 5dcf6997c5a081edf1d95a6a373c6b08#C#compaction#298 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-16 00:55:03,889 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/.tmp/C/235a4c8879424e8b8d3ca472badd0106 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/C/235a4c8879424e8b8d3ca472badd0106 2023-04-16 00:55:03,892 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 8 (all) file(s) in 5dcf6997c5a081edf1d95a6a373c6b08/C of 5dcf6997c5a081edf1d95a6a373c6b08 into 235a4c8879424e8b8d3ca472badd0106(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 00:55:03,892 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5dcf6997c5a081edf1d95a6a373c6b08: 2023-04-16 00:55:03,892 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08., storeName=5dcf6997c5a081edf1d95a6a373c6b08/C, priority=8, startTime=1681606503811; duration=0sec 2023-04-16 00:55:03,892 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 00:55:04,122 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(9158): Flush requested on 5dcf6997c5a081edf1d95a6a373c6b08 2023-04-16 00:55:04,122 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. as already flushing 2023-04-16 00:55:04,182 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681606504156, max timestamp=1681606504180Num uniques 859; 2023-04-16 00:55:04,182 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681606504156, max timestamp=1681606504180Num uniques 859; 2023-04-16 00:55:04,182 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681606504156, max timestamp=1681606504180Num uniques 859; 2023-04-16 00:55:04,183 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-16 00:55:04,182 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-16 00:55:04,183 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-16 00:55:04,239 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:04,239 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 378 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58292 deadline: 1681606564236, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:04,243 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/.tmp/B/d2110a183d2e4a2d81041aa71b5340ae as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/B/d2110a183d2e4a2d81041aa71b5340ae 2023-04-16 00:55:04,246 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HStore(1652): Completed compaction of 8 (all) file(s) in 5dcf6997c5a081edf1d95a6a373c6b08/B of 5dcf6997c5a081edf1d95a6a373c6b08 into d2110a183d2e4a2d81041aa71b5340ae(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 00:55:04,246 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5dcf6997c5a081edf1d95a6a373c6b08: 2023-04-16 00:55:04,246 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08., storeName=5dcf6997c5a081edf1d95a6a373c6b08/B, priority=8, startTime=1681606503811; duration=0sec 2023-04-16 00:55:04,247 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 00:55:04,279 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=396 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/.tmp/B/49afbc0b6d95439ba2891e384844b8dd 2023-04-16 00:55:04,288 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=396 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/.tmp/C/e5cfbe196b9040eea2d00a207298048b 2023-04-16 00:55:04,292 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/.tmp/A/5ad752265917432ca1117f1013767530 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/A/5ad752265917432ca1117f1013767530 2023-04-16 00:55:04,295 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/A/5ad752265917432ca1117f1013767530, entries=100, sequenceid=396, filesize=9.5 K 2023-04-16 00:55:04,296 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/.tmp/B/49afbc0b6d95439ba2891e384844b8dd as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/B/49afbc0b6d95439ba2891e384844b8dd 2023-04-16 00:55:04,300 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/B/49afbc0b6d95439ba2891e384844b8dd, entries=100, sequenceid=396, filesize=9.5 K 2023-04-16 00:55:04,301 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/.tmp/C/e5cfbe196b9040eea2d00a207298048b as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/C/e5cfbe196b9040eea2d00a207298048b 2023-04-16 00:55:04,305 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/C/e5cfbe196b9040eea2d00a207298048b, entries=100, sequenceid=396, filesize=9.5 K 2023-04-16 00:55:04,306 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 5dcf6997c5a081edf1d95a6a373c6b08 in 454ms, sequenceid=396, compaction requested=false 2023-04-16 00:55:04,306 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 5dcf6997c5a081edf1d95a6a373c6b08: 2023-04-16 00:55:04,306 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. 2023-04-16 00:55:04,306 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:55:04,306 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:55:04,306 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:04,306 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:55:04,306 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:55:04,308 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:04,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:04,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:04,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:04,308 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:04,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:04,308 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:04,308 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:55:04,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:04,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:04,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:04,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:04,310 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:55:04,310 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@e68499a[Count = 0] remaining members to acquire global barrier 2023-04-16 00:55:04,310 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:55:04,310 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:04,311 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:04,311 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:04,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:04,311 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:55:04,311 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:55:04,311 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:55:04,311 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:04,311 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:55:04,313 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:55:04,313 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:04,313 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:04,313 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:55:04,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:04,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:04,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:04,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:04,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:04,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:04,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:04,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:04,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:04,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:04,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:55:04,316 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:55:04,316 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:55:04,316 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:55:04,316 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:55:04,316 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:55:04,316 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:55:04,318 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:04,318 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:04,318 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:04,318 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:04,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:04,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:04,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:04,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:04,318 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:04,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:04,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:04,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:04,319 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:04,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:04,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:04,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:04,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:04,320 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:04,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:04,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:04,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:04,323 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:04,323 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:04,323 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:04,323 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:04,323 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:04,323 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:04,323 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:04,323 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:04,323 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:04,323 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:55:04,323 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:55:04,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:04,323 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:04,323 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:04,324 DEBUG [Thread-1736] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:55:04,324 DEBUG [Thread-1736] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:55:04,328 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:04,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:04,342 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(9158): Flush requested on 5dcf6997c5a081edf1d95a6a373c6b08 2023-04-16 00:55:04,342 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 5dcf6997c5a081edf1d95a6a373c6b08 3/3 column families, dataSize=134.18 KB heapSize=384.98 KB 2023-04-16 00:55:04,342 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5dcf6997c5a081edf1d95a6a373c6b08, store=A 2023-04-16 00:55:04,342 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 00:55:04,343 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5dcf6997c5a081edf1d95a6a373c6b08, store=B 2023-04-16 00:55:04,343 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 00:55:04,343 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5dcf6997c5a081edf1d95a6a373c6b08, store=C 2023-04-16 00:55:04,343 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 00:55:04,366 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:04,366 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 388 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58292 deadline: 1681606564364, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:04,424 DEBUG [Thread-1736] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:55:04,424 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:55:04,426 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:55:04,426 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:55:04,427 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:55:04,427 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:55:04,427 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:04,427 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:55:04,427 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:55:04,427 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:04,427 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:04,428 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:04,428 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:04,428 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:04,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:04,428 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:04,428 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:55:04,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:04,429 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:04,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:55:04,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:04,429 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:55:04,430 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:55:04,430 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:55:04,430 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:55:04,430 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:04,430 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:55:04,430 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:55:04,430 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:55:04,430 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. 2023-04-16 00:55:04,430 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. started... 2023-04-16 00:55:04,430 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. as already flushing 2023-04-16 00:55:04,430 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. 2023-04-16 00:55:04,430 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:55:04,431 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:55:04,431 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:04,431 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:55:04,431 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:55:04,433 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:04,433 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:04,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:04,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:04,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:04,434 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:04,434 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:55:04,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:04,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:04,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:04,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:04,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:04,435 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:55:04,435 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@67c2e977[Count = 0] remaining members to acquire global barrier 2023-04-16 00:55:04,435 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:55:04,435 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:04,436 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:04,436 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:04,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:04,436 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:55:04,436 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:55:04,436 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:55:04,436 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:04,436 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:55:04,438 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:55:04,438 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:04,438 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:04,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:04,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:04,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:04,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:04,438 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:55:04,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:04,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:04,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:04,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:04,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:04,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:04,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:55:04,440 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:55:04,440 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:55:04,440 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:55:04,440 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:55:04,440 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:55:04,440 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:55:04,447 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:04,447 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:04,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:04,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:04,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:04,448 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:04,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:04,448 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:04,448 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:04,448 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:04,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:04,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:04,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:04,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:04,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:04,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:04,450 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:04,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:04,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:04,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:04,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:04,451 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:04,451 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:04,451 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:04,451 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:04,451 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:55:04,451 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:04,451 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:55:04,452 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:04,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:04,452 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:04,451 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:04,451 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:04,452 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:04,452 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:04,452 DEBUG [Thread-1736] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:55:04,452 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:55:04,452 DEBUG [Thread-1736] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:55:04,453 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:04,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:04,468 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:04,469 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 390 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58292 deadline: 1681606564467, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:04,552 DEBUG [Thread-1736] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:55:04,553 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:55:04,554 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:55:04,555 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:55:04,555 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:55:04,555 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:55:04,555 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:04,555 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:55:04,555 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:55:04,556 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:04,556 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:04,557 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:04,557 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:04,557 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:04,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:04,557 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:04,557 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:55:04,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:04,558 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:04,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:55:04,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:04,558 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:55:04,558 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:55:04,558 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:55:04,558 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:55:04,558 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:04,558 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:55:04,558 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:55:04,558 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:55:04,558 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. 2023-04-16 00:55:04,559 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. started... 2023-04-16 00:55:04,559 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. as already flushing 2023-04-16 00:55:04,559 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. 2023-04-16 00:55:04,559 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:55:04,559 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:55:04,559 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:04,559 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:55:04,559 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:55:04,560 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:04,560 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:04,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:04,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:04,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:04,561 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:04,561 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:55:04,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:04,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:04,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:04,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:04,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:04,562 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:55:04,562 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5dd7d9a3[Count = 0] remaining members to acquire global barrier 2023-04-16 00:55:04,562 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:55:04,562 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:04,565 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:04,565 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:04,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:04,565 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:55:04,565 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:55:04,565 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:55:04,565 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:04,565 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:55:04,566 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:55:04,566 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:04,566 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:04,566 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:55:04,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:04,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:04,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:04,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:04,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:04,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:04,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:04,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:04,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:04,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:04,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:55:04,568 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:55:04,568 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:55:04,569 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:55:04,569 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:55:04,569 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:55:04,569 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:55:04,570 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:04,570 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:04,570 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:04,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:04,570 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:04,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:04,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:04,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:04,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:04,571 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:04,571 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:04,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:04,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:04,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:04,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:04,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:04,572 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:04,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:04,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:04,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:04,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:04,574 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:04,574 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:04,574 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:04,574 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:04,574 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:04,574 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:04,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:04,574 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:04,574 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:55:04,574 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:55:04,574 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:55:04,574 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:04,574 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:04,574 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:04,574 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:04,575 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:04,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:04,575 DEBUG [Thread-1736] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:55:04,575 DEBUG [Thread-1736] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:55:04,672 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:04,672 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 392 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58292 deadline: 1681606564670, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:04,675 DEBUG [Thread-1736] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:55:04,676 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:55:04,677 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:55:04,678 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:55:04,678 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:55:04,678 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:55:04,678 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:04,678 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:55:04,678 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:55:04,679 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:04,679 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:04,681 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:04,681 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:04,681 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:04,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:04,682 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:04,682 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:55:04,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:04,682 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:04,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:55:04,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:04,683 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:55:04,683 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:55:04,683 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:55:04,683 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:55:04,683 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:04,683 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:55:04,684 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:55:04,684 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:55:04,684 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. 2023-04-16 00:55:04,684 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. started... 2023-04-16 00:55:04,684 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. as already flushing 2023-04-16 00:55:04,684 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. 2023-04-16 00:55:04,684 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:55:04,684 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:55:04,684 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:04,684 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:55:04,684 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:55:04,685 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:04,685 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:04,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:04,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:04,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:04,686 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:04,686 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:55:04,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:04,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:04,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:04,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:04,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:04,687 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:55:04,688 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@26fb4321[Count = 0] remaining members to acquire global barrier 2023-04-16 00:55:04,688 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:55:04,688 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:04,689 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:04,689 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:04,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:04,689 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:55:04,689 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:55:04,689 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:55:04,689 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:04,689 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:55:04,691 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:04,691 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:55:04,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:04,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:04,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:04,691 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:04,691 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:55:04,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:04,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:04,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:04,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:04,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:04,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:04,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:04,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:55:04,694 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:55:04,694 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:55:04,694 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:55:04,694 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:55:04,694 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:55:04,694 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:55:04,697 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:04,697 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:04,697 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:04,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:04,697 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:04,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:04,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:04,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:04,697 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:04,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:04,697 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:04,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:04,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:04,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:04,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:04,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:04,698 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:04,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:04,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:04,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:04,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:04,700 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:04,700 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:04,700 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:04,700 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:04,700 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:04,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:04,700 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:04,700 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:04,700 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:55:04,701 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:55:04,701 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:55:04,700 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:04,701 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:04,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:04,701 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:04,701 DEBUG [Thread-1736] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:55:04,701 DEBUG [Thread-1736] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:55:04,701 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:04,701 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:04,751 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=435 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/.tmp/A/63e53f23f2d04ce9a07cac76d72140f0 2023-04-16 00:55:04,759 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=435 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/.tmp/B/07562d43a3744788a6306617d615f5b1 2023-04-16 00:55:04,801 DEBUG [Thread-1736] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:55:04,802 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:55:04,803 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:55:04,804 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:55:04,804 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:55:04,804 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:55:04,804 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:04,804 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:55:04,804 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:55:04,805 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:04,805 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:04,806 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:04,806 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:04,806 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:04,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:04,806 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:04,806 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:55:04,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:04,807 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:04,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:55:04,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:04,807 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:55:04,808 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:55:04,808 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:55:04,808 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:55:04,808 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:04,808 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:55:04,808 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:55:04,808 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:55:04,808 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. 2023-04-16 00:55:04,808 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. started... 2023-04-16 00:55:04,808 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. as already flushing 2023-04-16 00:55:04,808 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. 2023-04-16 00:55:04,808 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:55:04,808 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:55:04,808 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:04,808 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:55:04,808 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:55:04,810 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:04,810 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:04,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:04,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:04,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:04,810 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:04,810 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:55:04,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:04,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:04,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:04,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:04,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:04,811 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:55:04,811 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6a13eafd[Count = 0] remaining members to acquire global barrier 2023-04-16 00:55:04,811 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:55:04,811 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:04,813 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:04,813 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:04,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:04,814 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:55:04,814 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:04,814 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:55:04,814 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:55:04,814 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:55:04,815 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:04,815 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:55:04,815 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:04,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:04,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:04,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:04,815 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:55:04,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:04,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:04,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:04,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:04,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:04,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:04,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:04,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:55:04,817 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:55:04,817 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:55:04,817 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:55:04,817 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:55:04,818 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:55:04,818 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:55:04,819 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:04,819 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:04,819 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:04,820 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:04,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:04,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:04,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:04,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:04,820 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:04,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:04,820 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:04,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:04,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:04,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:04,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:04,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:04,821 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:04,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:04,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:04,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:04,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:04,826 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:04,826 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:04,826 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:04,826 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:55:04,826 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:55:04,826 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:04,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:04,826 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:04,827 DEBUG [Thread-1736] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:55:04,827 DEBUG [Thread-1736] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:55:04,826 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:04,827 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:04,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:04,826 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:04,827 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:55:04,827 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:04,827 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:04,827 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:04,827 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:04,927 DEBUG [Thread-1736] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:55:04,927 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:55:04,929 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:55:04,930 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:55:04,930 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:55:04,930 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:55:04,930 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:04,930 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:55:04,930 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:55:04,931 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:04,931 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:04,932 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:04,932 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:04,932 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:04,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:04,932 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:04,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:04,932 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:55:04,933 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:04,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:55:04,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:04,933 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:55:04,933 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:55:04,933 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:55:04,933 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:55:04,933 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:04,933 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:55:04,934 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:55:04,934 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:55:04,934 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. 2023-04-16 00:55:04,934 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. started... 2023-04-16 00:55:04,934 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. as already flushing 2023-04-16 00:55:04,934 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. 2023-04-16 00:55:04,934 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:55:04,934 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:55:04,934 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:04,934 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:55:04,934 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:55:04,935 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:04,936 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:04,936 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:04,936 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:55:04,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:04,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:04,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:04,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:04,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:04,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:04,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:04,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:04,937 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:55:04,938 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1a6a3f99[Count = 0] remaining members to acquire global barrier 2023-04-16 00:55:04,938 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:55:04,938 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:04,939 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:04,939 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:04,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:04,939 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:55:04,939 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:55:04,939 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:04,939 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:55:04,939 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:55:04,941 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:55:04,942 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:04,941 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:04,942 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:55:04,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:04,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:04,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:04,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:04,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:04,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:04,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:04,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:04,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:04,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:04,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:55:04,944 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:55:04,944 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:55:04,944 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:55:04,944 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:55:04,944 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:55:04,944 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:55:04,945 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:04,945 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:04,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:04,946 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:04,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:04,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:04,946 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:04,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:04,946 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:04,946 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:04,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:04,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:04,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:04,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:04,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:04,947 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:04,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:04,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:04,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:04,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:04,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:04,949 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:04,949 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:04,949 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:04,949 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:55:04,949 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:55:04,949 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:04,949 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:04,949 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:04,949 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:04,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:04,950 DEBUG [Thread-1736] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:55:04,950 DEBUG [Thread-1736] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:55:04,949 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:04,950 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:04,949 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:04,950 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:55:04,950 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:04,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:04,950 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:04,975 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:04,976 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 394 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58292 deadline: 1681606564975, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:05,050 DEBUG [Thread-1736] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:55:05,050 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:55:05,052 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:55:05,053 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:55:05,053 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:55:05,053 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:55:05,053 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:05,053 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:55:05,053 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:55:05,053 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:05,053 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:05,054 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:05,055 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:05,055 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:05,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:05,055 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:05,055 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:55:05,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:05,056 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:05,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:55:05,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:05,056 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:55:05,056 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:55:05,057 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:55:05,057 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:55:05,057 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:05,057 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:55:05,057 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:55:05,057 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:55:05,057 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. 2023-04-16 00:55:05,057 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. started... 2023-04-16 00:55:05,057 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. as already flushing 2023-04-16 00:55:05,057 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. 2023-04-16 00:55:05,057 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:55:05,057 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:55:05,058 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:05,058 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:55:05,058 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:55:05,060 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:05,060 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:05,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:05,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:05,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:05,060 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:05,060 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:55:05,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:05,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:05,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:05,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:05,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:05,062 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:55:05,062 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7913f7b6[Count = 0] remaining members to acquire global barrier 2023-04-16 00:55:05,062 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:55:05,062 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:05,063 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:05,063 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:05,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:05,063 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:55:05,063 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:55:05,063 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:05,063 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:55:05,063 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:55:05,064 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:55:05,064 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:05,064 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:05,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:05,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:05,064 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:55:05,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:05,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:05,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:05,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:05,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:05,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:05,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:05,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:05,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:55:05,067 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:55:05,067 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:55:05,067 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:55:05,067 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:55:05,067 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:55:05,067 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:55:05,068 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:05,068 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:05,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:05,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:05,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:05,068 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:05,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:05,068 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:05,069 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:05,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:05,069 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:05,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:05,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:05,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:05,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:05,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:05,070 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:05,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:05,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:05,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:05,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:05,073 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:05,073 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:05,073 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:05,073 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:55:05,073 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:55:05,073 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:05,073 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:05,073 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:05,073 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:05,073 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:05,073 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:05,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:05,073 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:05,073 DEBUG [Thread-1736] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:55:05,074 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:05,074 DEBUG [Thread-1736] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:55:05,074 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:05,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:05,168 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=435 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/.tmp/C/23bf671eea75402b85751fcd56eefde4 2023-04-16 00:55:05,172 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/.tmp/A/63e53f23f2d04ce9a07cac76d72140f0 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/A/63e53f23f2d04ce9a07cac76d72140f0 2023-04-16 00:55:05,174 DEBUG [Thread-1736] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:55:05,174 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:55:05,176 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/A/63e53f23f2d04ce9a07cac76d72140f0, entries=200, sequenceid=435, filesize=14.3 K 2023-04-16 00:55:05,176 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:55:05,177 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/.tmp/B/07562d43a3744788a6306617d615f5b1 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/B/07562d43a3744788a6306617d615f5b1 2023-04-16 00:55:05,177 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:55:05,177 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:55:05,177 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:55:05,177 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:05,177 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:55:05,177 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:55:05,178 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:05,178 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:05,179 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:05,179 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:05,179 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:05,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:05,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:05,180 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:05,180 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:55:05,180 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:05,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:55:05,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:05,181 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:55:05,181 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/B/07562d43a3744788a6306617d615f5b1, entries=150, sequenceid=435, filesize=11.9 K 2023-04-16 00:55:05,182 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/.tmp/C/23bf671eea75402b85751fcd56eefde4 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/C/23bf671eea75402b85751fcd56eefde4 2023-04-16 00:55:05,187 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:55:05,187 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:55:05,187 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:05,187 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:55:05,187 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:55:05,188 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:55:05,188 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. 2023-04-16 00:55:05,188 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. started... 2023-04-16 00:55:05,188 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. as already flushing 2023-04-16 00:55:05,188 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. 2023-04-16 00:55:05,188 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:55:05,188 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:55:05,188 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:05,188 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:55:05,188 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:55:05,189 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/C/23bf671eea75402b85751fcd56eefde4, entries=150, sequenceid=435, filesize=11.9 K 2023-04-16 00:55:05,189 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:05,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:05,189 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:05,190 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~384.94 KB/394176, currentSize=46.96 KB/48090 for 5dcf6997c5a081edf1d95a6a373c6b08 in 848ms, sequenceid=435, compaction requested=true 2023-04-16 00:55:05,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:05,190 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 5dcf6997c5a081edf1d95a6a373c6b08: 2023-04-16 00:55:05,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:05,190 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 00:55:05,190 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:05,190 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:55:05,190 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 00:55:05,190 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-16 00:55:05,190 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 00:55:05,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:05,190 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-16 00:55:05,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:05,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:05,191 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37440 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-16 00:55:05,191 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HStore(1912): 5dcf6997c5a081edf1d95a6a373c6b08/A is initiating minor compaction (all files) 2023-04-16 00:55:05,191 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 5dcf6997c5a081edf1d95a6a373c6b08/A in TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. 2023-04-16 00:55:05,191 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/A/ac924eb5b02647e3b3347a8c6e8dfd70, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/A/5ad752265917432ca1117f1013767530, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/A/63e53f23f2d04ce9a07cac76d72140f0] into tmpdir=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/.tmp, totalSize=36.6 K 2023-04-16 00:55:05,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:05,192 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35000 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-16 00:55:05,192 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.Compactor(207): Compacting ac924eb5b02647e3b3347a8c6e8dfd70, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=384, earliestPutTs=1681606502350 2023-04-16 00:55:05,192 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HStore(1912): 5dcf6997c5a081edf1d95a6a373c6b08/B is initiating minor compaction (all files) 2023-04-16 00:55:05,192 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 5dcf6997c5a081edf1d95a6a373c6b08/B in TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. 2023-04-16 00:55:05,192 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/B/d2110a183d2e4a2d81041aa71b5340ae, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/B/49afbc0b6d95439ba2891e384844b8dd, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/B/07562d43a3744788a6306617d615f5b1] into tmpdir=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/.tmp, totalSize=34.2 K 2023-04-16 00:55:05,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:05,192 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.Compactor(207): Compacting 5ad752265917432ca1117f1013767530, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=396, earliestPutTs=1681606502984 2023-04-16 00:55:05,192 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:55:05,193 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:55:05,193 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:05,193 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting d2110a183d2e4a2d81041aa71b5340ae, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=384, earliestPutTs=1681606502350 2023-04-16 00:55:05,193 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@58d41fe[Count = 0] remaining members to acquire global barrier 2023-04-16 00:55:05,194 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting 49afbc0b6d95439ba2891e384844b8dd, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=396, earliestPutTs=1681606502984 2023-04-16 00:55:05,194 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:05,194 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:05,194 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:55:05,194 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.Compactor(207): Compacting 63e53f23f2d04ce9a07cac76d72140f0, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=435, earliestPutTs=1681606504217 2023-04-16 00:55:05,194 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting 07562d43a3744788a6306617d615f5b1, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=435, earliestPutTs=1681606504217 2023-04-16 00:55:05,194 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:05,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:05,196 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:55:05,196 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:55:05,196 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:55:05,199 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:05,199 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:55:05,199 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:05,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:05,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:05,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:05,199 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:55:05,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:05,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:05,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:05,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:05,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:05,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:05,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:05,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:55:05,202 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:55:05,202 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:55:05,202 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:55:05,202 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:55:05,203 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:55:05,203 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:55:05,204 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:05,204 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:05,204 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:05,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:05,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:05,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:05,204 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:05,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:05,205 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:05,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:05,205 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:05,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:05,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:05,205 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] throttle.PressureAwareThroughputController(145): 5dcf6997c5a081edf1d95a6a373c6b08#A#compaction#303 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-16 00:55:05,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:05,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:05,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:05,206 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:05,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:05,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:05,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:05,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:05,211 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:05,211 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:05,211 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:05,211 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:55:05,212 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:55:05,212 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:55:05,212 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:05,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:05,211 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:05,211 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:05,211 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:05,212 DEBUG [Thread-1736] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:55:05,212 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:05,212 DEBUG [Thread-1736] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:55:05,212 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:05,212 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:05,212 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:05,213 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] throttle.PressureAwareThroughputController(145): 5dcf6997c5a081edf1d95a6a373c6b08#B#compaction#304 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-16 00:55:05,213 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:05,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:05,232 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/.tmp/B/514f5956c0e548db94e19bf6c49ebc31 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/B/514f5956c0e548db94e19bf6c49ebc31 2023-04-16 00:55:05,234 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/.tmp/A/467bcd61a4794222924fb8e5a4e9a623 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/A/467bcd61a4794222924fb8e5a4e9a623 2023-04-16 00:55:05,239 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 5dcf6997c5a081edf1d95a6a373c6b08/B of 5dcf6997c5a081edf1d95a6a373c6b08 into 514f5956c0e548db94e19bf6c49ebc31(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 00:55:05,239 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5dcf6997c5a081edf1d95a6a373c6b08: 2023-04-16 00:55:05,239 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 5dcf6997c5a081edf1d95a6a373c6b08/A of 5dcf6997c5a081edf1d95a6a373c6b08 into 467bcd61a4794222924fb8e5a4e9a623(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 00:55:05,240 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5dcf6997c5a081edf1d95a6a373c6b08: 2023-04-16 00:55:05,240 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08., storeName=5dcf6997c5a081edf1d95a6a373c6b08/A, priority=13, startTime=1681606505190; duration=0sec 2023-04-16 00:55:05,240 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 00:55:05,240 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-16 00:55:05,239 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08., storeName=5dcf6997c5a081edf1d95a6a373c6b08/B, priority=13, startTime=1681606505190; duration=0sec 2023-04-16 00:55:05,240 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 00:55:05,241 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35000 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-16 00:55:05,241 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HStore(1912): 5dcf6997c5a081edf1d95a6a373c6b08/C is initiating minor compaction (all files) 2023-04-16 00:55:05,241 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 5dcf6997c5a081edf1d95a6a373c6b08/C in TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. 2023-04-16 00:55:05,241 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/C/235a4c8879424e8b8d3ca472badd0106, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/C/e5cfbe196b9040eea2d00a207298048b, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/C/23bf671eea75402b85751fcd56eefde4] into tmpdir=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/.tmp, totalSize=34.2 K 2023-04-16 00:55:05,241 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.Compactor(207): Compacting 235a4c8879424e8b8d3ca472badd0106, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=384, earliestPutTs=1681606502350 2023-04-16 00:55:05,241 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.Compactor(207): Compacting e5cfbe196b9040eea2d00a207298048b, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=396, earliestPutTs=1681606502984 2023-04-16 00:55:05,242 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.Compactor(207): Compacting 23bf671eea75402b85751fcd56eefde4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=435, earliestPutTs=1681606504217 2023-04-16 00:55:05,254 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] throttle.PressureAwareThroughputController(145): 5dcf6997c5a081edf1d95a6a373c6b08#C#compaction#305 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-16 00:55:05,312 DEBUG [Thread-1736] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:55:05,313 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:55:05,314 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:55:05,315 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:55:05,315 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:55:05,315 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:55:05,315 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:05,315 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:55:05,315 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:55:05,316 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:05,316 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:05,318 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:05,318 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:05,318 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:05,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:05,318 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:05,318 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:55:05,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:05,319 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:05,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:55:05,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:05,319 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:55:05,319 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:55:05,319 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:55:05,319 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:55:05,319 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:05,319 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:55:05,319 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:55:05,319 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:55:05,319 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. 2023-04-16 00:55:05,319 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. started... 2023-04-16 00:55:05,320 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 5dcf6997c5a081edf1d95a6a373c6b08 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-16 00:55:05,320 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5dcf6997c5a081edf1d95a6a373c6b08, store=A 2023-04-16 00:55:05,320 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:55:05,320 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5dcf6997c5a081edf1d95a6a373c6b08, store=B 2023-04-16 00:55:05,320 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:55:05,320 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5dcf6997c5a081edf1d95a6a373c6b08, store=C 2023-04-16 00:55:05,320 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:55:05,479 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(9158): Flush requested on 5dcf6997c5a081edf1d95a6a373c6b08 2023-04-16 00:55:05,479 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. as already flushing 2023-04-16 00:55:05,523 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681606505505, max timestamp=1681606505522Num uniques 859; 2023-04-16 00:55:05,523 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-16 00:55:05,524 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681606505505, max timestamp=1681606505522Num uniques 859; 2023-04-16 00:55:05,524 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681606505505, max timestamp=1681606505522Num uniques 859; 2023-04-16 00:55:05,524 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-16 00:55:05,524 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-16 00:55:05,568 DEBUG [Thread-1737] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x08cd7cb0 to 127.0.0.1:53150 2023-04-16 00:55:05,568 DEBUG [Thread-1730] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3157d4df to 127.0.0.1:53150 2023-04-16 00:55:05,568 DEBUG [Thread-1741] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x433f251d to 127.0.0.1:53150 2023-04-16 00:55:05,568 DEBUG [Thread-1737] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-16 00:55:05,568 DEBUG [Thread-1741] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-16 00:55:05,568 DEBUG [Thread-1730] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-16 00:55:05,570 DEBUG [Thread-1745] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3b4658f8 to 127.0.0.1:53150 2023-04-16 00:55:05,570 DEBUG [Thread-1745] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-16 00:55:05,570 DEBUG [Thread-1739] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x0ea1ca03 to 127.0.0.1:53150 2023-04-16 00:55:05,570 DEBUG [Thread-1743] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x66ee0bd0 to 127.0.0.1:53150 2023-04-16 00:55:05,570 DEBUG [Thread-1739] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-16 00:55:05,570 DEBUG [Thread-1743] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-16 00:55:05,661 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/.tmp/C/55d6e03f98664c1489856bc9a2c5de41 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/C/55d6e03f98664c1489856bc9a2c5de41 2023-04-16 00:55:05,665 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 5dcf6997c5a081edf1d95a6a373c6b08/C of 5dcf6997c5a081edf1d95a6a373c6b08 into 55d6e03f98664c1489856bc9a2c5de41(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 00:55:05,665 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5dcf6997c5a081edf1d95a6a373c6b08: 2023-04-16 00:55:05,665 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08., storeName=5dcf6997c5a081edf1d95a6a373c6b08/C, priority=13, startTime=1681606505190; duration=0sec 2023-04-16 00:55:05,665 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 00:55:05,727 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=447 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/.tmp/A/2ddd2069a1264328958fcbe98997eaed 2023-04-16 00:55:06,135 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=447 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/.tmp/B/84816833ae62402a84b25e0ff994eba0 2023-04-16 00:55:06,544 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=447 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/.tmp/C/bd3c0a94d38846cea9000fb2daa87263 2023-04-16 00:55:06,547 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/.tmp/A/2ddd2069a1264328958fcbe98997eaed as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/A/2ddd2069a1264328958fcbe98997eaed 2023-04-16 00:55:06,550 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/A/2ddd2069a1264328958fcbe98997eaed, entries=150, sequenceid=447, filesize=11.9 K 2023-04-16 00:55:06,550 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/.tmp/B/84816833ae62402a84b25e0ff994eba0 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/B/84816833ae62402a84b25e0ff994eba0 2023-04-16 00:55:06,553 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/B/84816833ae62402a84b25e0ff994eba0, entries=150, sequenceid=447, filesize=11.9 K 2023-04-16 00:55:06,554 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/.tmp/C/bd3c0a94d38846cea9000fb2daa87263 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/C/bd3c0a94d38846cea9000fb2daa87263 2023-04-16 00:55:06,557 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/C/bd3c0a94d38846cea9000fb2daa87263, entries=150, sequenceid=447, filesize=11.9 K 2023-04-16 00:55:06,557 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=127.47 KB/130530 for 5dcf6997c5a081edf1d95a6a373c6b08 in 1237ms, sequenceid=447, compaction requested=false 2023-04-16 00:55:06,558 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 5dcf6997c5a081edf1d95a6a373c6b08: 2023-04-16 00:55:06,558 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. 2023-04-16 00:55:06,558 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:55:06,558 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:55:06,558 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:06,558 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:55:06,558 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:55:06,559 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:06,559 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:06,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:06,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:06,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:06,559 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:06,560 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:55:06,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:06,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:06,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:06,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:06,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:06,561 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:55:06,561 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@703c17b0[Count = 0] remaining members to acquire global barrier 2023-04-16 00:55:06,561 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:55:06,561 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:06,563 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:06,563 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:06,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:06,563 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:55:06,563 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:55:06,563 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:55:06,563 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:06,563 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:55:06,564 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:55:06,564 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:06,564 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:06,565 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:55:06,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:06,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:06,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:06,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:06,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:06,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:06,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:06,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:06,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:06,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:06,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:55:06,566 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:55:06,566 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:55:06,566 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:55:06,566 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:55:06,567 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:55:06,567 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:55:06,568 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:06,568 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:06,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:06,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:06,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:06,568 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:06,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:06,568 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:06,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:06,568 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:06,568 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:06,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:06,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:06,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:06,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:06,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:06,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:06,569 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:06,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:06,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:06,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:06,571 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:06,571 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:06,571 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:06,571 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:06,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:06,571 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:06,571 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:55:06,571 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:55:06,571 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:06,571 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:06,571 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:06,571 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:06,571 DEBUG [Thread-1736] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:55:06,571 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:06,571 DEBUG [Thread-1736] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:55:06,571 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:06,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:06,571 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:06,671 DEBUG [Thread-1736] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:55:06,672 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:55:09,446 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(9158): Flush requested on 5dcf6997c5a081edf1d95a6a373c6b08 2023-04-16 00:55:09,446 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 5dcf6997c5a081edf1d95a6a373c6b08 3/3 column families, dataSize=134.18 KB heapSize=384.98 KB 2023-04-16 00:55:09,446 DEBUG [Thread-1726] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x23c97092 to 127.0.0.1:53150 2023-04-16 00:55:09,446 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5dcf6997c5a081edf1d95a6a373c6b08, store=A 2023-04-16 00:55:09,446 DEBUG [Thread-1726] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-16 00:55:09,446 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 00:55:09,446 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5dcf6997c5a081edf1d95a6a373c6b08, store=B 2023-04-16 00:55:09,446 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 00:55:09,446 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5dcf6997c5a081edf1d95a6a373c6b08, store=C 2023-04-16 00:55:09,446 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 00:55:09,506 DEBUG [Thread-1732] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3cf5bdb7 to 127.0.0.1:53150 2023-04-16 00:55:09,506 DEBUG [Thread-1732] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-16 00:55:09,516 DEBUG [Thread-1734] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6919b2a0 to 127.0.0.1:53150 2023-04-16 00:55:09,516 DEBUG [Thread-1734] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-16 00:55:09,540 DEBUG [Thread-1728] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5c124820 to 127.0.0.1:53150 2023-04-16 00:55:09,540 DEBUG [Thread-1728] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-16 00:55:09,540 INFO [Listener at localhost/33477] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-16 00:55:09,541 INFO [Listener at localhost/33477] hbase.AcidGuaranteesTestTool(394): wrote 20 2023-04-16 00:55:09,541 INFO [Listener at localhost/33477] hbase.AcidGuaranteesTestTool(394): wrote 23 2023-04-16 00:55:09,541 INFO [Listener at localhost/33477] hbase.AcidGuaranteesTestTool(394): wrote 303 2023-04-16 00:55:09,541 INFO [Listener at localhost/33477] hbase.AcidGuaranteesTestTool(394): wrote 22 2023-04-16 00:55:09,541 INFO [Listener at localhost/33477] hbase.AcidGuaranteesTestTool(394): wrote 22 2023-04-16 00:55:09,541 INFO [Listener at localhost/33477] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-16 00:55:09,541 INFO [Listener at localhost/33477] hbase.AcidGuaranteesTestTool(398): read 5420 2023-04-16 00:55:09,541 INFO [Listener at localhost/33477] hbase.AcidGuaranteesTestTool(398): read 5501 2023-04-16 00:55:09,541 INFO [Listener at localhost/33477] hbase.AcidGuaranteesTestTool(398): read 5402 2023-04-16 00:55:09,541 INFO [Listener at localhost/33477] hbase.AcidGuaranteesTestTool(398): read 5457 2023-04-16 00:55:09,541 INFO [Listener at localhost/33477] hbase.AcidGuaranteesTestTool(398): read 5511 2023-04-16 00:55:09,541 INFO [Listener at localhost/33477] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-16 00:55:09,541 INFO [Listener at localhost/33477] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-16 00:55:09,541 DEBUG [Listener at localhost/33477] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x12e82edd to 127.0.0.1:53150 2023-04-16 00:55:09,541 DEBUG [Listener at localhost/33477] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-16 00:55:09,542 INFO [Listener at localhost/33477] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-16 00:55:09,542 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable TestAcidGuarantees 2023-04-16 00:55:09,543 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure2.ProcedureExecutor(1029): Stored pid=31, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-16 00:55:09,545 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-16 00:55:09,545 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681606509545"}]},"ts":"1681606509545"} 2023-04-16 00:55:09,546 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-16 00:55:09,547 INFO [PEWorker-2] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-16 00:55:09,547 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=32, ppid=31, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=5dcf6997c5a081edf1d95a6a373c6b08, UNASSIGN}] 2023-04-16 00:55:09,549 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=32, ppid=31, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=5dcf6997c5a081edf1d95a6a373c6b08, UNASSIGN 2023-04-16 00:55:09,550 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=32 updating hbase:meta row=5dcf6997c5a081edf1d95a6a373c6b08, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:09,550 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681606509550"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681606509550"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681606509550"}]},"ts":"1681606509550"} 2023-04-16 00:55:09,550 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=33, ppid=32, state=RUNNABLE; CloseRegionProcedure 5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217}] 2023-04-16 00:55:09,645 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-16 00:55:09,702 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 5dcf6997c5a081edf1d95a6a373c6b08 2023-04-16 00:55:09,702 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 5dcf6997c5a081edf1d95a6a373c6b08, disabling compactions & flushes 2023-04-16 00:55:09,702 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1858): waiting for 0 compactions & cache flush to complete for region TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. 2023-04-16 00:55:09,846 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-16 00:55:09,853 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=485 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/.tmp/A/a1eeb191c3af46fe99eb60eeb2f65145 2023-04-16 00:55:10,147 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-16 00:55:10,262 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=485 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/.tmp/B/f7a89ad5cdcf48f3895b6c07ce934e27 2023-04-16 00:55:10,647 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-16 00:55:10,671 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=485 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/.tmp/C/d780baada4d748b9b429a28e12ef5ad4 2023-04-16 00:55:10,674 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/.tmp/A/a1eeb191c3af46fe99eb60eeb2f65145 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/A/a1eeb191c3af46fe99eb60eeb2f65145 2023-04-16 00:55:10,677 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/A/a1eeb191c3af46fe99eb60eeb2f65145, entries=150, sequenceid=485, filesize=11.9 K 2023-04-16 00:55:10,678 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/.tmp/B/f7a89ad5cdcf48f3895b6c07ce934e27 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/B/f7a89ad5cdcf48f3895b6c07ce934e27 2023-04-16 00:55:10,680 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/B/f7a89ad5cdcf48f3895b6c07ce934e27, entries=150, sequenceid=485, filesize=11.9 K 2023-04-16 00:55:10,681 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/.tmp/C/d780baada4d748b9b429a28e12ef5ad4 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/C/d780baada4d748b9b429a28e12ef5ad4 2023-04-16 00:55:10,684 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/C/d780baada4d748b9b429a28e12ef5ad4, entries=150, sequenceid=485, filesize=11.9 K 2023-04-16 00:55:10,685 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~384.94 KB/394176, currentSize=20.13 KB/20610 for 5dcf6997c5a081edf1d95a6a373c6b08 in 1239ms, sequenceid=485, compaction requested=true 2023-04-16 00:55:10,685 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 5dcf6997c5a081edf1d95a6a373c6b08: 2023-04-16 00:55:10,685 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. 2023-04-16 00:55:10,685 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 00:55:10,685 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. because compaction request was cancelled 2023-04-16 00:55:10,685 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 00:55:10,685 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. because compaction request was cancelled 2023-04-16 00:55:10,685 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 00:55:10,685 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. 2023-04-16 00:55:10,685 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. because compaction request was cancelled 2023-04-16 00:55:10,685 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. after waiting 0 ms 2023-04-16 00:55:10,685 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. 2023-04-16 00:55:10,685 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 5dcf6997c5a081edf1d95a6a373c6b08 3/3 column families, dataSize=20.13 KB heapSize=60.52 KB 2023-04-16 00:55:10,685 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5dcf6997c5a081edf1d95a6a373c6b08, store=A 2023-04-16 00:55:10,685 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:55:10,685 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5dcf6997c5a081edf1d95a6a373c6b08, store=B 2023-04-16 00:55:10,685 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:55:10,685 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5dcf6997c5a081edf1d95a6a373c6b08, store=C 2023-04-16 00:55:10,685 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:55:11,091 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=491 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/.tmp/A/c93dba6e3c084ce989bc1296abef79ba 2023-04-16 00:55:11,500 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=491 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/.tmp/B/2f7b6b2b35b64052b3392d24987eae6b 2023-04-16 00:55:11,648 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-16 00:55:11,909 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=491 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/.tmp/C/da3f1c54b5634531a366eef1fc3722e6 2023-04-16 00:55:11,912 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/.tmp/A/c93dba6e3c084ce989bc1296abef79ba as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/A/c93dba6e3c084ce989bc1296abef79ba 2023-04-16 00:55:11,915 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/A/c93dba6e3c084ce989bc1296abef79ba, entries=50, sequenceid=491, filesize=7.1 K 2023-04-16 00:55:11,916 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/.tmp/B/2f7b6b2b35b64052b3392d24987eae6b as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/B/2f7b6b2b35b64052b3392d24987eae6b 2023-04-16 00:55:11,919 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/B/2f7b6b2b35b64052b3392d24987eae6b, entries=50, sequenceid=491, filesize=7.1 K 2023-04-16 00:55:11,919 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/.tmp/C/da3f1c54b5634531a366eef1fc3722e6 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/C/da3f1c54b5634531a366eef1fc3722e6 2023-04-16 00:55:11,922 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/C/da3f1c54b5634531a366eef1fc3722e6, entries=50, sequenceid=491, filesize=7.1 K 2023-04-16 00:55:11,923 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~20.13 KB/20610, heapSize ~60.47 KB/61920, currentSize=0 B/0 for 5dcf6997c5a081edf1d95a6a373c6b08 in 1238ms, sequenceid=491, compaction requested=true 2023-04-16 00:55:11,941 DEBUG [StoreCloser-TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/A/234b83427e2c4b31b199e2d41ca9a8b7, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/A/763d2b70147c46289187c95232169c9e, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/A/455293189bd74ec48814096fbe126c40, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/A/3bcea7d9776f4f0b81265429ea0ef32f, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/A/5da8050d3e274b71a9e194c9cb082c26, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/A/d100f6235c09495fa58b0f7fa41a0a28, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/A/ddfda75fb7554390b8b938dca019fdaa, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/A/d4e6bc4b26f145a5a1616ea22cc689c2, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/A/96b5fc95057744639455f4bc8f1fa666, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/A/14ae83c08fcc4fa58a3b06de5e07b03c, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/A/3ce09f4850a94551ab78a00899f062b9, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/A/21e01d66e41248dca66286b70b732221, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/A/395a461575f54884afaddaeac4a27715, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/A/594881b111294649973ffe9db9eba08d, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/A/16d91c09d2e246e289467d4e502b812a, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/A/3b716faa19c94eed94906c9cb3cc426e, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/A/493a39147c7e4c09a8e6b6cc2f778a71, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/A/ea063ba4012d48b7ac186c0b20fc87ea, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/A/0cfe3148d92049a1914960bb03abdc6c, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/A/3b6431bd48ec4e368d2fb70f1a3ee38d, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/A/92b52e941fd741bf99e8e0603a09364b, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/A/f8f606d88e564c66ace684989cfa2ce1, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/A/bfa1e1144893410a9611e4123990091a, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/A/8d979d19aa2640b4b8d71ecf55b7c38f, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/A/695a75e780414782951c6fd010c6cea6, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/A/d59b4c0a106c4062b8ac41f6b4f0f578, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/A/ac924eb5b02647e3b3347a8c6e8dfd70, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/A/5ad752265917432ca1117f1013767530, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/A/63e53f23f2d04ce9a07cac76d72140f0] to archive 2023-04-16 00:55:11,942 DEBUG [StoreCloser-TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-16 00:55:11,943 DEBUG [StoreCloser-TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/A/234b83427e2c4b31b199e2d41ca9a8b7 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/A/234b83427e2c4b31b199e2d41ca9a8b7 2023-04-16 00:55:11,944 DEBUG [StoreCloser-TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/A/763d2b70147c46289187c95232169c9e to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/A/763d2b70147c46289187c95232169c9e 2023-04-16 00:55:11,945 DEBUG [StoreCloser-TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/A/455293189bd74ec48814096fbe126c40 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/A/455293189bd74ec48814096fbe126c40 2023-04-16 00:55:11,946 DEBUG [StoreCloser-TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/A/3bcea7d9776f4f0b81265429ea0ef32f to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/A/3bcea7d9776f4f0b81265429ea0ef32f 2023-04-16 00:55:11,947 DEBUG [StoreCloser-TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/A/5da8050d3e274b71a9e194c9cb082c26 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/A/5da8050d3e274b71a9e194c9cb082c26 2023-04-16 00:55:11,948 DEBUG [StoreCloser-TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/A/d100f6235c09495fa58b0f7fa41a0a28 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/A/d100f6235c09495fa58b0f7fa41a0a28 2023-04-16 00:55:11,948 DEBUG [StoreCloser-TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/A/ddfda75fb7554390b8b938dca019fdaa to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/A/ddfda75fb7554390b8b938dca019fdaa 2023-04-16 00:55:11,949 DEBUG [StoreCloser-TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/A/d4e6bc4b26f145a5a1616ea22cc689c2 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/A/d4e6bc4b26f145a5a1616ea22cc689c2 2023-04-16 00:55:11,950 DEBUG [StoreCloser-TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/A/96b5fc95057744639455f4bc8f1fa666 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/A/96b5fc95057744639455f4bc8f1fa666 2023-04-16 00:55:11,951 DEBUG [StoreCloser-TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/A/14ae83c08fcc4fa58a3b06de5e07b03c to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/A/14ae83c08fcc4fa58a3b06de5e07b03c 2023-04-16 00:55:11,951 DEBUG [StoreCloser-TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/A/3ce09f4850a94551ab78a00899f062b9 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/A/3ce09f4850a94551ab78a00899f062b9 2023-04-16 00:55:11,952 DEBUG [StoreCloser-TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/A/21e01d66e41248dca66286b70b732221 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/A/21e01d66e41248dca66286b70b732221 2023-04-16 00:55:11,953 DEBUG [StoreCloser-TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/A/395a461575f54884afaddaeac4a27715 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/A/395a461575f54884afaddaeac4a27715 2023-04-16 00:55:11,954 DEBUG [StoreCloser-TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/A/594881b111294649973ffe9db9eba08d to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/A/594881b111294649973ffe9db9eba08d 2023-04-16 00:55:11,955 DEBUG [StoreCloser-TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/A/16d91c09d2e246e289467d4e502b812a to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/A/16d91c09d2e246e289467d4e502b812a 2023-04-16 00:55:11,955 DEBUG [StoreCloser-TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/A/3b716faa19c94eed94906c9cb3cc426e to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/A/3b716faa19c94eed94906c9cb3cc426e 2023-04-16 00:55:11,956 DEBUG [StoreCloser-TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/A/493a39147c7e4c09a8e6b6cc2f778a71 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/A/493a39147c7e4c09a8e6b6cc2f778a71 2023-04-16 00:55:11,957 DEBUG [StoreCloser-TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/A/ea063ba4012d48b7ac186c0b20fc87ea to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/A/ea063ba4012d48b7ac186c0b20fc87ea 2023-04-16 00:55:11,958 DEBUG [StoreCloser-TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/A/0cfe3148d92049a1914960bb03abdc6c to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/A/0cfe3148d92049a1914960bb03abdc6c 2023-04-16 00:55:11,959 DEBUG [StoreCloser-TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/A/3b6431bd48ec4e368d2fb70f1a3ee38d to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/A/3b6431bd48ec4e368d2fb70f1a3ee38d 2023-04-16 00:55:11,960 DEBUG [StoreCloser-TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/A/92b52e941fd741bf99e8e0603a09364b to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/A/92b52e941fd741bf99e8e0603a09364b 2023-04-16 00:55:11,960 DEBUG [StoreCloser-TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/A/f8f606d88e564c66ace684989cfa2ce1 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/A/f8f606d88e564c66ace684989cfa2ce1 2023-04-16 00:55:11,961 DEBUG [StoreCloser-TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/A/bfa1e1144893410a9611e4123990091a to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/A/bfa1e1144893410a9611e4123990091a 2023-04-16 00:55:11,962 DEBUG [StoreCloser-TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/A/8d979d19aa2640b4b8d71ecf55b7c38f to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/A/8d979d19aa2640b4b8d71ecf55b7c38f 2023-04-16 00:55:11,963 DEBUG [StoreCloser-TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/A/695a75e780414782951c6fd010c6cea6 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/A/695a75e780414782951c6fd010c6cea6 2023-04-16 00:55:11,964 DEBUG [StoreCloser-TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/A/d59b4c0a106c4062b8ac41f6b4f0f578 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/A/d59b4c0a106c4062b8ac41f6b4f0f578 2023-04-16 00:55:11,964 DEBUG [StoreCloser-TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/A/ac924eb5b02647e3b3347a8c6e8dfd70 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/A/ac924eb5b02647e3b3347a8c6e8dfd70 2023-04-16 00:55:11,965 DEBUG [StoreCloser-TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/A/5ad752265917432ca1117f1013767530 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/A/5ad752265917432ca1117f1013767530 2023-04-16 00:55:11,966 DEBUG [StoreCloser-TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/A/63e53f23f2d04ce9a07cac76d72140f0 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/A/63e53f23f2d04ce9a07cac76d72140f0 2023-04-16 00:55:11,969 DEBUG [StoreCloser-TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/B/1a1b8c00fba64695895917834d3a77c5, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/B/261b40f4e6f44095baa2081690ff7c51, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/B/d66b1d97395345d0a463b6b3ed8ebe0a, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/B/0f1483715d8c4ee096ca03b10a1ccd11, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/B/1699ebb87e4844ca9710c20f85a137ee, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/B/cb9ce6a037de40c282327f598b27e035, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/B/df50817c07ac46c1a3a80441a2bfdbf3, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/B/ffec08e54d9c42539282ed2f59c3a937, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/B/720dc9d7aa6645c4a9c332a498a4e86d, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/B/fa08ea119d3b4da48d98ea0fa4dbf149, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/B/c4ca91c939e447aa983f1eebb0c4eccf, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/B/7a2472cd8eac43c9a09282a5adbfdfdd, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/B/d2b8e5a9229d4239b3501a67e0bf4653, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/B/8104b6ceeda84d92a329a2713a104504, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/B/5cf2c5d02ae249d7955278274d8aa969, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/B/e07b3cf10a3947bdb6e75b9f55a3abe9, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/B/ad79c85f7ba949e189083c0e3c80155f, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/B/b69c4c09edf74bb19ff8957d7ddfb6f3, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/B/137031767256439991bf77d0ea6987a2, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/B/11e14a78f35741d385ec8ab39d209a8e, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/B/409cf632708e4f4abd31f556b844e326, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/B/c467234f3865490e819ebb8a275995e7, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/B/1f91bba7117f441797d27bd90fe7b530, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/B/601f847e752342c7859b48241d444303, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/B/1cd1e91e9e6b4fa38b9e9a0ac3b2bc94, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/B/d2110a183d2e4a2d81041aa71b5340ae, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/B/5b9326a81a6a4ac086028ac1e417f543, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/B/49afbc0b6d95439ba2891e384844b8dd, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/B/07562d43a3744788a6306617d615f5b1] to archive 2023-04-16 00:55:11,969 DEBUG [StoreCloser-TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-16 00:55:11,970 DEBUG [StoreCloser-TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/B/1a1b8c00fba64695895917834d3a77c5 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/B/1a1b8c00fba64695895917834d3a77c5 2023-04-16 00:55:11,971 DEBUG [StoreCloser-TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/B/261b40f4e6f44095baa2081690ff7c51 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/B/261b40f4e6f44095baa2081690ff7c51 2023-04-16 00:55:11,972 DEBUG [StoreCloser-TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/B/d66b1d97395345d0a463b6b3ed8ebe0a to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/B/d66b1d97395345d0a463b6b3ed8ebe0a 2023-04-16 00:55:11,973 DEBUG [StoreCloser-TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/B/0f1483715d8c4ee096ca03b10a1ccd11 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/B/0f1483715d8c4ee096ca03b10a1ccd11 2023-04-16 00:55:11,974 DEBUG [StoreCloser-TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/B/1699ebb87e4844ca9710c20f85a137ee to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/B/1699ebb87e4844ca9710c20f85a137ee 2023-04-16 00:55:11,975 DEBUG [StoreCloser-TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/B/cb9ce6a037de40c282327f598b27e035 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/B/cb9ce6a037de40c282327f598b27e035 2023-04-16 00:55:11,976 DEBUG [StoreCloser-TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/B/df50817c07ac46c1a3a80441a2bfdbf3 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/B/df50817c07ac46c1a3a80441a2bfdbf3 2023-04-16 00:55:11,977 DEBUG [StoreCloser-TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/B/ffec08e54d9c42539282ed2f59c3a937 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/B/ffec08e54d9c42539282ed2f59c3a937 2023-04-16 00:55:11,977 DEBUG [StoreCloser-TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/B/720dc9d7aa6645c4a9c332a498a4e86d to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/B/720dc9d7aa6645c4a9c332a498a4e86d 2023-04-16 00:55:11,978 DEBUG [StoreCloser-TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/B/fa08ea119d3b4da48d98ea0fa4dbf149 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/B/fa08ea119d3b4da48d98ea0fa4dbf149 2023-04-16 00:55:11,979 DEBUG [StoreCloser-TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/B/c4ca91c939e447aa983f1eebb0c4eccf to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/B/c4ca91c939e447aa983f1eebb0c4eccf 2023-04-16 00:55:11,980 DEBUG [StoreCloser-TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/B/7a2472cd8eac43c9a09282a5adbfdfdd to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/B/7a2472cd8eac43c9a09282a5adbfdfdd 2023-04-16 00:55:11,981 DEBUG [StoreCloser-TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/B/d2b8e5a9229d4239b3501a67e0bf4653 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/B/d2b8e5a9229d4239b3501a67e0bf4653 2023-04-16 00:55:11,981 DEBUG [StoreCloser-TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/B/8104b6ceeda84d92a329a2713a104504 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/B/8104b6ceeda84d92a329a2713a104504 2023-04-16 00:55:11,982 DEBUG [StoreCloser-TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/B/5cf2c5d02ae249d7955278274d8aa969 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/B/5cf2c5d02ae249d7955278274d8aa969 2023-04-16 00:55:11,983 DEBUG [StoreCloser-TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/B/e07b3cf10a3947bdb6e75b9f55a3abe9 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/B/e07b3cf10a3947bdb6e75b9f55a3abe9 2023-04-16 00:55:11,984 DEBUG [StoreCloser-TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/B/ad79c85f7ba949e189083c0e3c80155f to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/B/ad79c85f7ba949e189083c0e3c80155f 2023-04-16 00:55:11,985 DEBUG [StoreCloser-TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/B/b69c4c09edf74bb19ff8957d7ddfb6f3 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/B/b69c4c09edf74bb19ff8957d7ddfb6f3 2023-04-16 00:55:11,986 DEBUG [StoreCloser-TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/B/137031767256439991bf77d0ea6987a2 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/B/137031767256439991bf77d0ea6987a2 2023-04-16 00:55:11,986 DEBUG [StoreCloser-TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/B/11e14a78f35741d385ec8ab39d209a8e to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/B/11e14a78f35741d385ec8ab39d209a8e 2023-04-16 00:55:11,987 DEBUG [StoreCloser-TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/B/409cf632708e4f4abd31f556b844e326 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/B/409cf632708e4f4abd31f556b844e326 2023-04-16 00:55:11,988 DEBUG [StoreCloser-TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/B/c467234f3865490e819ebb8a275995e7 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/B/c467234f3865490e819ebb8a275995e7 2023-04-16 00:55:11,989 DEBUG [StoreCloser-TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/B/1f91bba7117f441797d27bd90fe7b530 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/B/1f91bba7117f441797d27bd90fe7b530 2023-04-16 00:55:11,990 DEBUG [StoreCloser-TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/B/601f847e752342c7859b48241d444303 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/B/601f847e752342c7859b48241d444303 2023-04-16 00:55:11,991 DEBUG [StoreCloser-TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/B/1cd1e91e9e6b4fa38b9e9a0ac3b2bc94 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/B/1cd1e91e9e6b4fa38b9e9a0ac3b2bc94 2023-04-16 00:55:11,992 DEBUG [StoreCloser-TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/B/d2110a183d2e4a2d81041aa71b5340ae to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/B/d2110a183d2e4a2d81041aa71b5340ae 2023-04-16 00:55:11,993 DEBUG [StoreCloser-TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/B/5b9326a81a6a4ac086028ac1e417f543 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/B/5b9326a81a6a4ac086028ac1e417f543 2023-04-16 00:55:11,994 DEBUG [StoreCloser-TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/B/49afbc0b6d95439ba2891e384844b8dd to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/B/49afbc0b6d95439ba2891e384844b8dd 2023-04-16 00:55:11,994 DEBUG [StoreCloser-TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/B/07562d43a3744788a6306617d615f5b1 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/B/07562d43a3744788a6306617d615f5b1 2023-04-16 00:55:11,997 DEBUG [StoreCloser-TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/C/90cf6ae6569f4bc6a55737127551399f, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/C/bc37fce4d91344a199d34a2b4d21ecec, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/C/45b7fbb66e384f82be01c400fa6913f0, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/C/5ad005f3206149d9827cf860b162acd9, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/C/afa1a8d0db9c4fcfa46a1e30a3bbb4c3, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/C/edbcee3cbb9c467aa1bc5a2d92765b97, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/C/fbddbfa531154e4d8c66f90278d2c90a, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/C/c11279046bc54aee9e0e44d59ebf98ac, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/C/96e1a998284c456aa0504ba6f47358e3, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/C/ad87da1fbdfa4b3e9215a86ce90d710a, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/C/b60e05604a59435fb573ac3f216f8a22, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/C/fb1258db8b4f49b192fc75ab9e45bbbc, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/C/e1e64f4f28a74feeb7c45332135a1ad7, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/C/b57ba0adbfda4c23a83653c5127981c8, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/C/817e1c997c3c4ca789942f7f4cc9bc2c, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/C/cd98e72955444c60813ffd6e967016b3, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/C/559f4f6d943e4ae887accca147a58da9, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/C/1fa18240d3e6447ba26dc86aa6049938, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/C/5d4247d06e8c4949a2c18ec9add67da4, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/C/78c60dd18be44cb7acca58b59ac7efcd, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/C/496709f960b244d79395d4a01ba31a9e, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/C/6b37490beafd4793a113780f46c9b8fe, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/C/4901da3437ca4446b7c7bcce0d7136d5, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/C/5c858daf872b4aa0bad320f785a9a9aa, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/C/4ff8907dd9bb49a29564c119a9b3421e, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/C/235a4c8879424e8b8d3ca472badd0106, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/C/523c8eeb3ed0490696a815142fa0bc3a, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/C/e5cfbe196b9040eea2d00a207298048b, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/C/23bf671eea75402b85751fcd56eefde4] to archive 2023-04-16 00:55:11,997 DEBUG [StoreCloser-TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-16 00:55:11,999 DEBUG [StoreCloser-TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/C/90cf6ae6569f4bc6a55737127551399f to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/C/90cf6ae6569f4bc6a55737127551399f 2023-04-16 00:55:12,000 DEBUG [StoreCloser-TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/C/bc37fce4d91344a199d34a2b4d21ecec to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/C/bc37fce4d91344a199d34a2b4d21ecec 2023-04-16 00:55:12,000 DEBUG [StoreCloser-TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/C/45b7fbb66e384f82be01c400fa6913f0 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/C/45b7fbb66e384f82be01c400fa6913f0 2023-04-16 00:55:12,001 DEBUG [StoreCloser-TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/C/5ad005f3206149d9827cf860b162acd9 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/C/5ad005f3206149d9827cf860b162acd9 2023-04-16 00:55:12,002 DEBUG [StoreCloser-TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/C/afa1a8d0db9c4fcfa46a1e30a3bbb4c3 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/C/afa1a8d0db9c4fcfa46a1e30a3bbb4c3 2023-04-16 00:55:12,003 DEBUG [StoreCloser-TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/C/edbcee3cbb9c467aa1bc5a2d92765b97 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/C/edbcee3cbb9c467aa1bc5a2d92765b97 2023-04-16 00:55:12,004 DEBUG [StoreCloser-TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/C/fbddbfa531154e4d8c66f90278d2c90a to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/C/fbddbfa531154e4d8c66f90278d2c90a 2023-04-16 00:55:12,005 DEBUG [StoreCloser-TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/C/c11279046bc54aee9e0e44d59ebf98ac to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/C/c11279046bc54aee9e0e44d59ebf98ac 2023-04-16 00:55:12,005 DEBUG [StoreCloser-TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/C/96e1a998284c456aa0504ba6f47358e3 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/C/96e1a998284c456aa0504ba6f47358e3 2023-04-16 00:55:12,006 DEBUG [StoreCloser-TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/C/ad87da1fbdfa4b3e9215a86ce90d710a to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/C/ad87da1fbdfa4b3e9215a86ce90d710a 2023-04-16 00:55:12,007 DEBUG [StoreCloser-TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/C/b60e05604a59435fb573ac3f216f8a22 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/C/b60e05604a59435fb573ac3f216f8a22 2023-04-16 00:55:12,008 DEBUG [StoreCloser-TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/C/fb1258db8b4f49b192fc75ab9e45bbbc to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/C/fb1258db8b4f49b192fc75ab9e45bbbc 2023-04-16 00:55:12,009 DEBUG [StoreCloser-TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/C/e1e64f4f28a74feeb7c45332135a1ad7 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/C/e1e64f4f28a74feeb7c45332135a1ad7 2023-04-16 00:55:12,010 DEBUG [StoreCloser-TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/C/b57ba0adbfda4c23a83653c5127981c8 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/C/b57ba0adbfda4c23a83653c5127981c8 2023-04-16 00:55:12,011 DEBUG [StoreCloser-TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/C/817e1c997c3c4ca789942f7f4cc9bc2c to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/C/817e1c997c3c4ca789942f7f4cc9bc2c 2023-04-16 00:55:12,011 DEBUG [StoreCloser-TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/C/cd98e72955444c60813ffd6e967016b3 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/C/cd98e72955444c60813ffd6e967016b3 2023-04-16 00:55:12,012 DEBUG [StoreCloser-TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/C/559f4f6d943e4ae887accca147a58da9 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/C/559f4f6d943e4ae887accca147a58da9 2023-04-16 00:55:12,013 DEBUG [StoreCloser-TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/C/1fa18240d3e6447ba26dc86aa6049938 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/C/1fa18240d3e6447ba26dc86aa6049938 2023-04-16 00:55:12,014 DEBUG [StoreCloser-TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/C/5d4247d06e8c4949a2c18ec9add67da4 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/C/5d4247d06e8c4949a2c18ec9add67da4 2023-04-16 00:55:12,014 DEBUG [StoreCloser-TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/C/78c60dd18be44cb7acca58b59ac7efcd to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/C/78c60dd18be44cb7acca58b59ac7efcd 2023-04-16 00:55:12,015 DEBUG [StoreCloser-TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/C/496709f960b244d79395d4a01ba31a9e to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/C/496709f960b244d79395d4a01ba31a9e 2023-04-16 00:55:12,016 DEBUG [StoreCloser-TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/C/6b37490beafd4793a113780f46c9b8fe to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/C/6b37490beafd4793a113780f46c9b8fe 2023-04-16 00:55:12,016 DEBUG [StoreCloser-TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/C/4901da3437ca4446b7c7bcce0d7136d5 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/C/4901da3437ca4446b7c7bcce0d7136d5 2023-04-16 00:55:12,017 DEBUG [StoreCloser-TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/C/5c858daf872b4aa0bad320f785a9a9aa to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/C/5c858daf872b4aa0bad320f785a9a9aa 2023-04-16 00:55:12,018 DEBUG [StoreCloser-TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/C/4ff8907dd9bb49a29564c119a9b3421e to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/C/4ff8907dd9bb49a29564c119a9b3421e 2023-04-16 00:55:12,018 DEBUG [StoreCloser-TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/C/235a4c8879424e8b8d3ca472badd0106 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/C/235a4c8879424e8b8d3ca472badd0106 2023-04-16 00:55:12,019 DEBUG [StoreCloser-TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/C/523c8eeb3ed0490696a815142fa0bc3a to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/C/523c8eeb3ed0490696a815142fa0bc3a 2023-04-16 00:55:12,020 DEBUG [StoreCloser-TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/C/e5cfbe196b9040eea2d00a207298048b to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/C/e5cfbe196b9040eea2d00a207298048b 2023-04-16 00:55:12,021 DEBUG [StoreCloser-TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/C/23bf671eea75402b85751fcd56eefde4 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/C/23bf671eea75402b85751fcd56eefde4 2023-04-16 00:55:12,024 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/recovered.edits/494.seqid, newMaxSeqId=494, maxSeqId=1 2023-04-16 00:55:12,025 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08. 2023-04-16 00:55:12,025 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 5dcf6997c5a081edf1d95a6a373c6b08: 2023-04-16 00:55:12,026 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 5dcf6997c5a081edf1d95a6a373c6b08 2023-04-16 00:55:12,027 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=32 updating hbase:meta row=5dcf6997c5a081edf1d95a6a373c6b08, regionState=CLOSED 2023-04-16 00:55:12,027 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681606512027"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681606512027"}]},"ts":"1681606512027"} 2023-04-16 00:55:12,029 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=33, resume processing ppid=32 2023-04-16 00:55:12,029 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=33, ppid=32, state=SUCCESS; CloseRegionProcedure 5dcf6997c5a081edf1d95a6a373c6b08, server=jenkins-hbase4.apache.org,34975,1681606419217 in 2.4780 sec 2023-04-16 00:55:12,030 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=32, resume processing ppid=31 2023-04-16 00:55:12,030 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=32, ppid=31, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=5dcf6997c5a081edf1d95a6a373c6b08, UNASSIGN in 2.4820 sec 2023-04-16 00:55:12,030 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681606512030"}]},"ts":"1681606512030"} 2023-04-16 00:55:12,031 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-16 00:55:12,032 INFO [PEWorker-2] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-16 00:55:12,034 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=31, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 2.4910 sec 2023-04-16 00:55:13,649 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-16 00:55:13,649 INFO [Listener at localhost/33477] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 31 completed 2023-04-16 00:55:13,649 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete TestAcidGuarantees 2023-04-16 00:55:13,650 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure2.ProcedureExecutor(1029): Stored pid=34, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-16 00:55:13,651 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=34, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-16 00:55:13,652 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1227): Checking to see if procedure is done pid=34 2023-04-16 00:55:13,652 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=34, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-16 00:55:13,655 DEBUG [HFileArchiver-7] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/.tmp/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08 2023-04-16 00:55:13,657 DEBUG [HFileArchiver-7] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/.tmp/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/A, FileablePath, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/.tmp/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/B, FileablePath, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/.tmp/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/C, FileablePath, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/.tmp/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/recovered.edits] 2023-04-16 00:55:13,660 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/.tmp/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/A/2ddd2069a1264328958fcbe98997eaed to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/A/2ddd2069a1264328958fcbe98997eaed 2023-04-16 00:55:13,661 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/.tmp/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/A/467bcd61a4794222924fb8e5a4e9a623 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/A/467bcd61a4794222924fb8e5a4e9a623 2023-04-16 00:55:13,661 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/.tmp/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/A/a1eeb191c3af46fe99eb60eeb2f65145 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/A/a1eeb191c3af46fe99eb60eeb2f65145 2023-04-16 00:55:13,662 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/.tmp/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/A/c93dba6e3c084ce989bc1296abef79ba to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/A/c93dba6e3c084ce989bc1296abef79ba 2023-04-16 00:55:13,664 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/.tmp/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/B/2f7b6b2b35b64052b3392d24987eae6b to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/B/2f7b6b2b35b64052b3392d24987eae6b 2023-04-16 00:55:13,665 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/.tmp/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/B/514f5956c0e548db94e19bf6c49ebc31 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/B/514f5956c0e548db94e19bf6c49ebc31 2023-04-16 00:55:13,666 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/.tmp/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/B/84816833ae62402a84b25e0ff994eba0 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/B/84816833ae62402a84b25e0ff994eba0 2023-04-16 00:55:13,666 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/.tmp/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/B/f7a89ad5cdcf48f3895b6c07ce934e27 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/B/f7a89ad5cdcf48f3895b6c07ce934e27 2023-04-16 00:55:13,668 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/.tmp/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/C/55d6e03f98664c1489856bc9a2c5de41 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/C/55d6e03f98664c1489856bc9a2c5de41 2023-04-16 00:55:13,669 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/.tmp/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/C/bd3c0a94d38846cea9000fb2daa87263 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/C/bd3c0a94d38846cea9000fb2daa87263 2023-04-16 00:55:13,670 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/.tmp/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/C/d780baada4d748b9b429a28e12ef5ad4 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/C/d780baada4d748b9b429a28e12ef5ad4 2023-04-16 00:55:13,671 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/.tmp/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/C/da3f1c54b5634531a366eef1fc3722e6 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/C/da3f1c54b5634531a366eef1fc3722e6 2023-04-16 00:55:13,673 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/.tmp/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/recovered.edits/494.seqid to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08/recovered.edits/494.seqid 2023-04-16 00:55:13,673 DEBUG [HFileArchiver-7] backup.HFileArchiver(596): Deleted hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/.tmp/data/default/TestAcidGuarantees/5dcf6997c5a081edf1d95a6a373c6b08 2023-04-16 00:55:13,673 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-16 00:55:13,675 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=34, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-16 00:55:13,676 WARN [PEWorker-5] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-16 00:55:13,677 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-16 00:55:13,678 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=34, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-16 00:55:13,678 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-16 00:55:13,678 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1681606513678"}]},"ts":"9223372036854775807"} 2023-04-16 00:55:13,680 INFO [PEWorker-5] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-16 00:55:13,680 DEBUG [PEWorker-5] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 5dcf6997c5a081edf1d95a6a373c6b08, NAME => 'TestAcidGuarantees,,1681606483401.5dcf6997c5a081edf1d95a6a373c6b08.', STARTKEY => '', ENDKEY => ''}] 2023-04-16 00:55:13,680 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-16 00:55:13,680 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1681606513680"}]},"ts":"9223372036854775807"} 2023-04-16 00:55:13,681 INFO [PEWorker-5] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-16 00:55:13,683 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(130): Finished pid=34, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-16 00:55:13,684 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=34, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 34 msec 2023-04-16 00:55:13,752 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1227): Checking to see if procedure is done pid=34 2023-04-16 00:55:13,753 INFO [Listener at localhost/33477] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 34 completed 2023-04-16 00:55:13,763 INFO [Listener at localhost/33477] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testGetAtomicity Thread=225 (was 229), OpenFileDescriptor=543 (was 545), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=305 (was 337), ProcessCount=171 (was 171), AvailableMemoryMB=753 (was 786) 2023-04-16 00:55:13,773 INFO [Listener at localhost/33477] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testMobScanAtomicity Thread=225, OpenFileDescriptor=543, MaxFileDescriptor=60000, SystemLoadAverage=305, ProcessCount=171, AvailableMemoryMB=752 2023-04-16 00:55:13,775 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-16 00:55:13,775 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-16 00:55:13,776 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure2.ProcedureExecutor(1029): Stored pid=35, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-16 00:55:13,777 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-16 00:55:13,777 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 35 2023-04-16 00:55:13,778 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-16 00:55:13,778 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-04-16 00:55:13,779 DEBUG [HFileArchiver-8] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/.tmp/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5 2023-04-16 00:55:13,779 DEBUG [HFileArchiver-8] backup.HFileArchiver(153): Directory hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/.tmp/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5 empty. 2023-04-16 00:55:13,779 DEBUG [HFileArchiver-8] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/.tmp/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5 2023-04-16 00:55:13,779 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-16 00:55:13,879 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-04-16 00:55:14,079 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-04-16 00:55:14,186 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-16 00:55:14,188 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => ee2f682ad6dd0fe8e91c021d54d3bdc5, NAME => 'TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/.tmp 2023-04-16 00:55:14,380 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-04-16 00:55:14,593 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-16 00:55:14,594 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing ee2f682ad6dd0fe8e91c021d54d3bdc5, disabling compactions & flushes 2023-04-16 00:55:14,594 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. 2023-04-16 00:55:14,594 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. 2023-04-16 00:55:14,594 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. after waiting 0 ms 2023-04-16 00:55:14,594 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. 2023-04-16 00:55:14,594 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. 2023-04-16 00:55:14,594 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for ee2f682ad6dd0fe8e91c021d54d3bdc5: 2023-04-16 00:55:14,596 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-16 00:55:14,597 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681606514596"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681606514596"}]},"ts":"1681606514596"} 2023-04-16 00:55:14,597 INFO [PEWorker-1] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-16 00:55:14,598 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-16 00:55:14,598 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681606514598"}]},"ts":"1681606514598"} 2023-04-16 00:55:14,599 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-16 00:55:14,602 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=36, ppid=35, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=ee2f682ad6dd0fe8e91c021d54d3bdc5, ASSIGN}] 2023-04-16 00:55:14,603 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=36, ppid=35, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=ee2f682ad6dd0fe8e91c021d54d3bdc5, ASSIGN 2023-04-16 00:55:14,604 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=36, ppid=35, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=ee2f682ad6dd0fe8e91c021d54d3bdc5, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,34975,1681606419217; forceNewPlan=false, retain=false 2023-04-16 00:55:14,755 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=36 updating hbase:meta row=ee2f682ad6dd0fe8e91c021d54d3bdc5, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:14,755 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681606514755"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681606514755"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681606514755"}]},"ts":"1681606514755"} 2023-04-16 00:55:14,756 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=37, ppid=36, state=RUNNABLE; OpenRegionProcedure ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217}] 2023-04-16 00:55:14,881 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-04-16 00:55:14,910 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. 2023-04-16 00:55:14,910 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => ee2f682ad6dd0fe8e91c021d54d3bdc5, NAME => 'TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5.', STARTKEY => '', ENDKEY => ''} 2023-04-16 00:55:14,911 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees ee2f682ad6dd0fe8e91c021d54d3bdc5 2023-04-16 00:55:14,911 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-16 00:55:14,911 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for ee2f682ad6dd0fe8e91c021d54d3bdc5 2023-04-16 00:55:14,911 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for ee2f682ad6dd0fe8e91c021d54d3bdc5 2023-04-16 00:55:14,912 INFO [StoreOpener-ee2f682ad6dd0fe8e91c021d54d3bdc5-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region ee2f682ad6dd0fe8e91c021d54d3bdc5 2023-04-16 00:55:14,913 DEBUG [StoreOpener-ee2f682ad6dd0fe8e91c021d54d3bdc5-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/A 2023-04-16 00:55:14,913 DEBUG [StoreOpener-ee2f682ad6dd0fe8e91c021d54d3bdc5-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/A 2023-04-16 00:55:14,913 INFO [StoreOpener-ee2f682ad6dd0fe8e91c021d54d3bdc5-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-16 00:55:14,913 INFO [StoreOpener-ee2f682ad6dd0fe8e91c021d54d3bdc5-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region ee2f682ad6dd0fe8e91c021d54d3bdc5 columnFamilyName A 2023-04-16 00:55:14,914 INFO [StoreOpener-ee2f682ad6dd0fe8e91c021d54d3bdc5-1] regionserver.HStore(310): Store=ee2f682ad6dd0fe8e91c021d54d3bdc5/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-16 00:55:14,914 INFO [StoreOpener-ee2f682ad6dd0fe8e91c021d54d3bdc5-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region ee2f682ad6dd0fe8e91c021d54d3bdc5 2023-04-16 00:55:14,915 DEBUG [StoreOpener-ee2f682ad6dd0fe8e91c021d54d3bdc5-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/B 2023-04-16 00:55:14,915 DEBUG [StoreOpener-ee2f682ad6dd0fe8e91c021d54d3bdc5-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/B 2023-04-16 00:55:14,915 INFO [StoreOpener-ee2f682ad6dd0fe8e91c021d54d3bdc5-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-16 00:55:14,915 INFO [StoreOpener-ee2f682ad6dd0fe8e91c021d54d3bdc5-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region ee2f682ad6dd0fe8e91c021d54d3bdc5 columnFamilyName B 2023-04-16 00:55:14,916 INFO [StoreOpener-ee2f682ad6dd0fe8e91c021d54d3bdc5-1] regionserver.HStore(310): Store=ee2f682ad6dd0fe8e91c021d54d3bdc5/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-16 00:55:14,916 INFO [StoreOpener-ee2f682ad6dd0fe8e91c021d54d3bdc5-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region ee2f682ad6dd0fe8e91c021d54d3bdc5 2023-04-16 00:55:14,917 DEBUG [StoreOpener-ee2f682ad6dd0fe8e91c021d54d3bdc5-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/C 2023-04-16 00:55:14,917 DEBUG [StoreOpener-ee2f682ad6dd0fe8e91c021d54d3bdc5-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/C 2023-04-16 00:55:14,917 INFO [StoreOpener-ee2f682ad6dd0fe8e91c021d54d3bdc5-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-16 00:55:14,917 INFO [StoreOpener-ee2f682ad6dd0fe8e91c021d54d3bdc5-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region ee2f682ad6dd0fe8e91c021d54d3bdc5 columnFamilyName C 2023-04-16 00:55:14,917 INFO [StoreOpener-ee2f682ad6dd0fe8e91c021d54d3bdc5-1] regionserver.HStore(310): Store=ee2f682ad6dd0fe8e91c021d54d3bdc5/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-16 00:55:14,917 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. 2023-04-16 00:55:14,918 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5 2023-04-16 00:55:14,918 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5 2023-04-16 00:55:14,920 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-16 00:55:14,920 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for ee2f682ad6dd0fe8e91c021d54d3bdc5 2023-04-16 00:55:14,922 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-16 00:55:14,922 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened ee2f682ad6dd0fe8e91c021d54d3bdc5; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=63812202, jitterRate=-0.0491240918636322}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-16 00:55:14,922 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for ee2f682ad6dd0fe8e91c021d54d3bdc5: 2023-04-16 00:55:14,923 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5., pid=37, masterSystemTime=1681606514907 2023-04-16 00:55:14,924 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. 2023-04-16 00:55:14,924 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. 2023-04-16 00:55:14,924 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=36 updating hbase:meta row=ee2f682ad6dd0fe8e91c021d54d3bdc5, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:14,925 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681606514924"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1681606514924"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1681606514924"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1681606514924"}]},"ts":"1681606514924"} 2023-04-16 00:55:14,926 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=37, resume processing ppid=36 2023-04-16 00:55:14,926 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=37, ppid=36, state=SUCCESS; OpenRegionProcedure ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 in 169 msec 2023-04-16 00:55:14,927 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=36, resume processing ppid=35 2023-04-16 00:55:14,927 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=36, ppid=35, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=ee2f682ad6dd0fe8e91c021d54d3bdc5, ASSIGN in 324 msec 2023-04-16 00:55:14,928 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-16 00:55:14,928 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681606514928"}]},"ts":"1681606514928"} 2023-04-16 00:55:14,928 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-16 00:55:14,930 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-16 00:55:14,931 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=35, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 1.1550 sec 2023-04-16 00:55:15,882 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-04-16 00:55:15,882 INFO [Listener at localhost/33477] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 35 completed 2023-04-16 00:55:15,882 DEBUG [Listener at localhost/33477] zookeeper.ReadOnlyZKClient(139): Connect 0x0fd3d54e to 127.0.0.1:53150 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-16 00:55:15,887 DEBUG [Listener at localhost/33477] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6434ce29, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-16 00:55:15,888 DEBUG [Listener at localhost/33477] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-16 00:55:15,890 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:59580, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-16 00:55:15,891 DEBUG [Listener at localhost/33477] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-16 00:55:15,892 INFO [RS-EventLoopGroup-1-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:37666, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-16 00:55:15,894 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-16 00:55:15,894 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.HMaster$12(2454): Client=jenkins//172.31.14.131 modify table TestAcidGuarantees from 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} to 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', MOB_THRESHOLD => '4', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-16 00:55:15,895 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure2.ProcedureExecutor(1029): Stored pid=38, state=RUNNABLE:MODIFY_TABLE_PREPARE; ModifyTableProcedure table=TestAcidGuarantees 2023-04-16 00:55:16,303 DEBUG [PEWorker-4] util.FSTableDescriptors(570): Wrote into hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-16 00:55:16,304 DEBUG [PEWorker-4] util.FSTableDescriptors(519): Deleted hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-16 00:55:16,304 INFO [PEWorker-4] util.FSTableDescriptors(293): Updated tableinfo=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-16 00:55:16,306 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=39, ppid=38, state=RUNNABLE:REOPEN_TABLE_REGIONS_GET_REGIONS; ReopenTableRegionsProcedure table=TestAcidGuarantees}] 2023-04-16 00:55:16,308 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=40, ppid=39, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=ee2f682ad6dd0fe8e91c021d54d3bdc5, REOPEN/MOVE}] 2023-04-16 00:55:16,309 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=40, ppid=39, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=ee2f682ad6dd0fe8e91c021d54d3bdc5, REOPEN/MOVE 2023-04-16 00:55:16,309 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=ee2f682ad6dd0fe8e91c021d54d3bdc5, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:16,310 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681606516309"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681606516309"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681606516309"}]},"ts":"1681606516309"} 2023-04-16 00:55:16,310 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=41, ppid=40, state=RUNNABLE; CloseRegionProcedure ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217}] 2023-04-16 00:55:16,462 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close ee2f682ad6dd0fe8e91c021d54d3bdc5 2023-04-16 00:55:16,463 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing ee2f682ad6dd0fe8e91c021d54d3bdc5, disabling compactions & flushes 2023-04-16 00:55:16,463 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. 2023-04-16 00:55:16,463 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. 2023-04-16 00:55:16,463 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. after waiting 0 ms 2023-04-16 00:55:16,463 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. 2023-04-16 00:55:16,466 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-04-16 00:55:16,467 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. 2023-04-16 00:55:16,467 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for ee2f682ad6dd0fe8e91c021d54d3bdc5: 2023-04-16 00:55:16,467 WARN [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(3509): Not adding moved region record: ee2f682ad6dd0fe8e91c021d54d3bdc5 to self. 2023-04-16 00:55:16,468 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed ee2f682ad6dd0fe8e91c021d54d3bdc5 2023-04-16 00:55:16,468 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=ee2f682ad6dd0fe8e91c021d54d3bdc5, regionState=CLOSED 2023-04-16 00:55:16,468 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681606516468"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681606516468"}]},"ts":"1681606516468"} 2023-04-16 00:55:16,470 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=41, resume processing ppid=40 2023-04-16 00:55:16,470 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=41, ppid=40, state=SUCCESS; CloseRegionProcedure ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 in 159 msec 2023-04-16 00:55:16,470 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=40, ppid=39, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=ee2f682ad6dd0fe8e91c021d54d3bdc5, REOPEN/MOVE; state=CLOSED, location=jenkins-hbase4.apache.org,34975,1681606419217; forceNewPlan=false, retain=true 2023-04-16 00:55:16,621 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=ee2f682ad6dd0fe8e91c021d54d3bdc5, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:16,621 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681606516621"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681606516621"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681606516621"}]},"ts":"1681606516621"} 2023-04-16 00:55:16,622 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=42, ppid=40, state=RUNNABLE; OpenRegionProcedure ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217}] 2023-04-16 00:55:16,776 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. 2023-04-16 00:55:16,776 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => ee2f682ad6dd0fe8e91c021d54d3bdc5, NAME => 'TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5.', STARTKEY => '', ENDKEY => ''} 2023-04-16 00:55:16,777 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees ee2f682ad6dd0fe8e91c021d54d3bdc5 2023-04-16 00:55:16,777 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-16 00:55:16,777 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for ee2f682ad6dd0fe8e91c021d54d3bdc5 2023-04-16 00:55:16,777 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for ee2f682ad6dd0fe8e91c021d54d3bdc5 2023-04-16 00:55:16,778 INFO [StoreOpener-ee2f682ad6dd0fe8e91c021d54d3bdc5-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region ee2f682ad6dd0fe8e91c021d54d3bdc5 2023-04-16 00:55:16,779 DEBUG [StoreOpener-ee2f682ad6dd0fe8e91c021d54d3bdc5-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/A 2023-04-16 00:55:16,779 DEBUG [StoreOpener-ee2f682ad6dd0fe8e91c021d54d3bdc5-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/A 2023-04-16 00:55:16,780 INFO [StoreOpener-ee2f682ad6dd0fe8e91c021d54d3bdc5-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-16 00:55:16,780 INFO [StoreOpener-ee2f682ad6dd0fe8e91c021d54d3bdc5-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region ee2f682ad6dd0fe8e91c021d54d3bdc5 columnFamilyName A 2023-04-16 00:55:16,781 INFO [StoreOpener-ee2f682ad6dd0fe8e91c021d54d3bdc5-1] regionserver.HStore(310): Store=ee2f682ad6dd0fe8e91c021d54d3bdc5/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-16 00:55:16,781 INFO [StoreOpener-ee2f682ad6dd0fe8e91c021d54d3bdc5-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region ee2f682ad6dd0fe8e91c021d54d3bdc5 2023-04-16 00:55:16,782 DEBUG [StoreOpener-ee2f682ad6dd0fe8e91c021d54d3bdc5-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/B 2023-04-16 00:55:16,782 DEBUG [StoreOpener-ee2f682ad6dd0fe8e91c021d54d3bdc5-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/B 2023-04-16 00:55:16,782 INFO [StoreOpener-ee2f682ad6dd0fe8e91c021d54d3bdc5-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-16 00:55:16,782 INFO [StoreOpener-ee2f682ad6dd0fe8e91c021d54d3bdc5-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region ee2f682ad6dd0fe8e91c021d54d3bdc5 columnFamilyName B 2023-04-16 00:55:16,783 INFO [StoreOpener-ee2f682ad6dd0fe8e91c021d54d3bdc5-1] regionserver.HStore(310): Store=ee2f682ad6dd0fe8e91c021d54d3bdc5/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-16 00:55:16,783 INFO [StoreOpener-ee2f682ad6dd0fe8e91c021d54d3bdc5-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region ee2f682ad6dd0fe8e91c021d54d3bdc5 2023-04-16 00:55:16,783 DEBUG [StoreOpener-ee2f682ad6dd0fe8e91c021d54d3bdc5-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/C 2023-04-16 00:55:16,783 DEBUG [StoreOpener-ee2f682ad6dd0fe8e91c021d54d3bdc5-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/C 2023-04-16 00:55:16,783 INFO [StoreOpener-ee2f682ad6dd0fe8e91c021d54d3bdc5-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-16 00:55:16,783 INFO [StoreOpener-ee2f682ad6dd0fe8e91c021d54d3bdc5-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region ee2f682ad6dd0fe8e91c021d54d3bdc5 columnFamilyName C 2023-04-16 00:55:16,784 INFO [StoreOpener-ee2f682ad6dd0fe8e91c021d54d3bdc5-1] regionserver.HStore(310): Store=ee2f682ad6dd0fe8e91c021d54d3bdc5/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-16 00:55:16,784 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. 2023-04-16 00:55:16,784 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5 2023-04-16 00:55:16,785 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5 2023-04-16 00:55:16,787 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-16 00:55:16,788 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for ee2f682ad6dd0fe8e91c021d54d3bdc5 2023-04-16 00:55:16,788 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened ee2f682ad6dd0fe8e91c021d54d3bdc5; next sequenceid=5; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=72470639, jitterRate=0.07989667356014252}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-16 00:55:16,788 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for ee2f682ad6dd0fe8e91c021d54d3bdc5: 2023-04-16 00:55:16,789 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5., pid=42, masterSystemTime=1681606516774 2023-04-16 00:55:16,790 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. 2023-04-16 00:55:16,790 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. 2023-04-16 00:55:16,790 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=ee2f682ad6dd0fe8e91c021d54d3bdc5, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:16,790 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681606516790"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1681606516790"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1681606516790"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1681606516790"}]},"ts":"1681606516790"} 2023-04-16 00:55:16,792 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=42, resume processing ppid=40 2023-04-16 00:55:16,792 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=42, ppid=40, state=SUCCESS; OpenRegionProcedure ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 in 169 msec 2023-04-16 00:55:16,793 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=40, resume processing ppid=39 2023-04-16 00:55:16,793 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=40, ppid=39, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=ee2f682ad6dd0fe8e91c021d54d3bdc5, REOPEN/MOVE in 484 msec 2023-04-16 00:55:16,795 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=39, resume processing ppid=38 2023-04-16 00:55:16,795 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=39, ppid=38, state=SUCCESS; ReopenTableRegionsProcedure table=TestAcidGuarantees in 488 msec 2023-04-16 00:55:16,796 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=38, state=SUCCESS; ModifyTableProcedure table=TestAcidGuarantees in 901 msec 2023-04-16 00:55:16,797 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1227): Checking to see if procedure is done pid=38 2023-04-16 00:55:16,798 DEBUG [Listener at localhost/33477] zookeeper.ReadOnlyZKClient(139): Connect 0x1a5ab074 to 127.0.0.1:53150 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-16 00:55:16,803 DEBUG [Listener at localhost/33477] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@bb27fd4, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-16 00:55:16,804 DEBUG [Listener at localhost/33477] zookeeper.ReadOnlyZKClient(139): Connect 0x0bcfc1e0 to 127.0.0.1:53150 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-16 00:55:16,807 DEBUG [Listener at localhost/33477] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@445919fd, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-16 00:55:16,807 DEBUG [Listener at localhost/33477] zookeeper.ReadOnlyZKClient(139): Connect 0x38341134 to 127.0.0.1:53150 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-16 00:55:16,811 DEBUG [Listener at localhost/33477] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4d010713, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-16 00:55:16,811 DEBUG [Listener at localhost/33477] zookeeper.ReadOnlyZKClient(139): Connect 0x71b5058b to 127.0.0.1:53150 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-16 00:55:16,814 DEBUG [Listener at localhost/33477] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@69100738, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-16 00:55:16,814 DEBUG [Listener at localhost/33477] zookeeper.ReadOnlyZKClient(139): Connect 0x3620ee35 to 127.0.0.1:53150 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-16 00:55:16,819 DEBUG [Listener at localhost/33477] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6f603c27, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-16 00:55:16,820 DEBUG [Listener at localhost/33477] zookeeper.ReadOnlyZKClient(139): Connect 0x7e01d5e5 to 127.0.0.1:53150 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-16 00:55:16,823 DEBUG [Listener at localhost/33477] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@10ee5f06, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-16 00:55:16,823 DEBUG [Listener at localhost/33477] zookeeper.ReadOnlyZKClient(139): Connect 0x51ccd8b7 to 127.0.0.1:53150 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-16 00:55:16,829 DEBUG [Listener at localhost/33477] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@21128196, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-16 00:55:16,829 DEBUG [Listener at localhost/33477] zookeeper.ReadOnlyZKClient(139): Connect 0x68a0e21a to 127.0.0.1:53150 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-16 00:55:16,834 DEBUG [Listener at localhost/33477] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5f9e86d9, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-16 00:55:16,834 DEBUG [Listener at localhost/33477] zookeeper.ReadOnlyZKClient(139): Connect 0x1ec12b02 to 127.0.0.1:53150 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-16 00:55:16,839 DEBUG [Listener at localhost/33477] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@711ebaf2, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-16 00:55:16,839 DEBUG [Listener at localhost/33477] zookeeper.ReadOnlyZKClient(139): Connect 0x1d7ba242 to 127.0.0.1:53150 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-16 00:55:16,846 DEBUG [Listener at localhost/33477] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@a7503d4, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-16 00:55:16,854 DEBUG [hconnection-0x397821aa-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-16 00:55:16,854 DEBUG [hconnection-0x34211721-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-16 00:55:16,854 DEBUG [hconnection-0x1047fcaa-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-16 00:55:16,855 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:55:16,855 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:59592, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-16 00:55:16,857 DEBUG [hconnection-0x7d025e3a-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-16 00:55:16,858 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:59616, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-16 00:55:16,863 DEBUG [hconnection-0x71fa59ef-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-16 00:55:16,863 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:59602, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-16 00:55:16,867 DEBUG [hconnection-0x734984b2-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-16 00:55:16,868 DEBUG [hconnection-0x70a06e9b-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-16 00:55:16,863 DEBUG [hconnection-0x15a4ecd-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-16 00:55:16,864 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:59608, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-16 00:55:16,868 DEBUG [hconnection-0x5651b708-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-16 00:55:16,869 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:59650, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-16 00:55:16,869 DEBUG [hconnection-0x479f5e24-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-16 00:55:16,872 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:59674, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-16 00:55:16,873 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:59632, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-16 00:55:16,873 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:59664, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-16 00:55:16,874 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:59642, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-16 00:55:16,874 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:59666, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-16 00:55:16,881 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(9158): Flush requested on ee2f682ad6dd0fe8e91c021d54d3bdc5 2023-04-16 00:55:16,881 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:55:16,881 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing ee2f682ad6dd0fe8e91c021d54d3bdc5 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-16 00:55:16,881 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:55:16,883 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ee2f682ad6dd0fe8e91c021d54d3bdc5, store=A 2023-04-16 00:55:16,883 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:55:16,883 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:55:16,883 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ee2f682ad6dd0fe8e91c021d54d3bdc5, store=B 2023-04-16 00:55:16,883 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:16,884 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:55:16,884 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:55:16,884 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:55:16,884 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ee2f682ad6dd0fe8e91c021d54d3bdc5, store=C 2023-04-16 00:55:16,884 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:55:16,887 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:16,887 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:16,888 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:16,888 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:16,888 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:16,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:16,894 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:16,894 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:55:16,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:16,897 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:16,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:55:16,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:16,898 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:55:16,899 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:55:16,899 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:55:16,899 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:55:16,900 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:16,900 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:55:16,905 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=409, cellSize=18725, totalHeapSize=42776, min timestamp=1681606516889, max timestamp=1681606516903Num uniques 859; 2023-04-16 00:55:16,905 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=18725, suffix heap size=117064,new segment heap size=42776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=409 2023-04-16 00:55:16,905 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=409, cellSize=18725, totalHeapSize=42776, min timestamp=1681606516889, max timestamp=1681606516903Num uniques 859; 2023-04-16 00:55:16,905 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=409, cellSize=18725, totalHeapSize=42776, min timestamp=1681606516889, max timestamp=1681606516903Num uniques 859; 2023-04-16 00:55:16,905 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=18725, suffix heap size=117064,new segment heap size=42776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=409 2023-04-16 00:55:16,905 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=18725, suffix heap size=117064,new segment heap size=42776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=409 2023-04-16 00:55:16,908 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:55:16,908 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:55:16,908 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. 2023-04-16 00:55:16,908 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. started... 2023-04-16 00:55:16,908 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. as already flushing 2023-04-16 00:55:16,908 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. 2023-04-16 00:55:16,908 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:55:16,908 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:55:16,908 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:16,914 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:55:16,914 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:55:16,916 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:16,916 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:16,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:16,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:16,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:16,916 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:16,916 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:55:16,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:16,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:16,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:16,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:16,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:16,918 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:55:16,918 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3b6178d9[Count = 0] remaining members to acquire global barrier 2023-04-16 00:55:16,918 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:55:16,918 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:16,920 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:16,920 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:16,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:16,920 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:55:16,920 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:55:16,920 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:55:16,920 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:16,920 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:55:16,921 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:55:16,921 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:16,921 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:16,921 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:55:16,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:16,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:16,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:16,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:16,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:16,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:16,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:16,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:16,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:16,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:16,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:55:16,925 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:55:16,925 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:55:16,925 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:55:16,925 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:55:16,925 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:55:16,925 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:16,925 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:55:16,925 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59592 deadline: 1681606576920, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:16,925 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:16,925 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59666 deadline: 1681606576922, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:16,926 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:16,926 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59642 deadline: 1681606576922, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:16,926 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:16,926 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59664 deadline: 1681606576924, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:16,926 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:16,926 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59650 deadline: 1681606576925, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:16,927 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:16,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:16,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:16,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:16,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:16,927 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:16,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:16,928 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:16,928 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:16,928 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:16,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:16,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:16,928 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:16,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:16,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:16,929 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:16,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:16,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:16,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:16,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:16,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:16,935 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:16,935 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:16,935 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:16,935 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:16,935 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:16,935 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:16,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:16,935 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:55:16,935 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:55:16,935 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:16,935 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:16,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:16,935 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:55:16,935 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:16,935 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:16,935 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:16,935 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:16,935 DEBUG [Thread-2414] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:55:16,935 DEBUG [Thread-2414] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:55:17,030 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:17,030 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59592 deadline: 1681606577026, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:17,030 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:17,030 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59666 deadline: 1681606577026, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:17,031 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:17,031 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59642 deadline: 1681606577027, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:17,031 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:17,031 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59664 deadline: 1681606577028, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:17,031 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:17,031 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59650 deadline: 1681606577028, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:17,036 DEBUG [Thread-2414] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:55:17,036 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:55:17,039 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:55:17,050 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:55:17,051 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:55:17,051 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:55:17,051 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:17,051 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:55:17,051 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:55:17,051 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:17,051 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:17,052 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:17,052 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:17,052 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:17,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:17,052 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:17,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:17,053 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:55:17,053 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:17,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:55:17,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:17,053 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:55:17,053 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:55:17,053 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:55:17,053 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:55:17,053 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:17,053 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:55:17,054 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:55:17,054 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:55:17,054 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. 2023-04-16 00:55:17,054 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. started... 2023-04-16 00:55:17,054 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. as already flushing 2023-04-16 00:55:17,054 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. 2023-04-16 00:55:17,054 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:55:17,054 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:55:17,054 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:17,054 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:55:17,054 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:55:17,055 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:17,055 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:17,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:17,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:17,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:17,056 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:17,056 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:55:17,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:17,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:17,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:17,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:17,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:17,058 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:55:17,058 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5da98644[Count = 0] remaining members to acquire global barrier 2023-04-16 00:55:17,058 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:55:17,058 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:17,059 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:17,059 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:17,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:17,059 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:55:17,059 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:55:17,059 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:55:17,059 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:17,059 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:55:17,061 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:55:17,061 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:17,061 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:55:17,062 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:17,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:17,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:17,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:17,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:17,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:17,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:17,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:17,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:17,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:17,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:17,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:55:17,064 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:55:17,064 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:55:17,064 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:55:17,064 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:55:17,064 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:55:17,064 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:55:17,067 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:17,067 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:17,067 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:17,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:17,067 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:17,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:17,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:17,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:17,067 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:17,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:17,067 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:17,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:17,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:17,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:17,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:17,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:17,070 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:17,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:17,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:17,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:17,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:17,072 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:17,072 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:17,072 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:17,072 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:17,073 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:17,073 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:17,073 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:17,073 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:17,073 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:17,073 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:17,073 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:55:17,073 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:55:17,073 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:17,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:17,073 DEBUG [Thread-2414] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:55:17,073 DEBUG [Thread-2414] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:55:17,073 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:55:17,073 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:17,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:17,173 DEBUG [Thread-2414] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:55:17,174 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:55:17,175 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:55:17,179 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:55:17,179 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:55:17,179 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:55:17,179 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:17,179 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:55:17,179 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:55:17,179 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:17,179 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:17,180 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:17,180 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:17,181 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:17,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:17,181 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:17,181 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:55:17,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:17,181 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:17,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:55:17,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:17,182 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:55:17,182 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:55:17,182 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:55:17,182 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:55:17,182 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:17,182 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:55:17,182 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:55:17,182 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:55:17,182 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. 2023-04-16 00:55:17,182 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. started... 2023-04-16 00:55:17,182 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. as already flushing 2023-04-16 00:55:17,182 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. 2023-04-16 00:55:17,182 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:55:17,182 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:55:17,182 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:17,182 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:55:17,182 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:55:17,184 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:17,185 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:17,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:17,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:17,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:17,185 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:17,185 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:55:17,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:17,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:17,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:17,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:17,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:17,186 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:55:17,186 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@32c4aebe[Count = 0] remaining members to acquire global barrier 2023-04-16 00:55:17,186 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:55:17,187 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:17,187 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:17,187 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:17,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:17,188 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:55:17,188 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:55:17,188 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:55:17,188 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:17,188 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:55:17,189 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:17,189 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:55:17,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:17,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:17,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:17,189 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:17,189 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:55:17,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:17,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:17,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:17,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:17,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:17,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:17,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:17,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:55:17,191 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:55:17,191 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:55:17,192 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:55:17,192 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:55:17,192 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:55:17,192 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:55:17,193 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:17,193 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:17,193 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:17,193 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:17,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:17,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:17,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:17,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:17,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:17,194 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:17,194 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:17,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:17,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:17,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:17,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:17,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:17,195 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:17,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:17,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:17,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:17,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:17,197 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:17,197 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:17,198 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:17,198 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:17,198 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:17,198 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:17,198 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:55:17,198 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:17,198 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:55:17,198 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:17,198 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:17,198 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:55:17,198 DEBUG [Thread-2414] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:55:17,198 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:17,198 DEBUG [Thread-2414] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:55:17,198 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:17,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:17,199 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:17,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:17,235 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:17,235 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59592 deadline: 1681606577231, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:17,235 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:17,235 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59642 deadline: 1681606577232, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:17,235 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:17,235 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59650 deadline: 1681606577233, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:17,235 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:17,235 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59666 deadline: 1681606577233, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:17,237 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:17,237 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59664 deadline: 1681606577233, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:17,298 DEBUG [Thread-2414] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:55:17,299 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:55:17,300 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:55:17,301 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:55:17,302 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:55:17,302 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:55:17,302 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:17,302 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:55:17,302 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:55:17,302 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:17,302 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:17,303 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:17,303 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:17,303 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:17,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:17,304 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:17,304 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:55:17,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:17,304 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:17,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:55:17,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:17,304 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:55:17,305 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:55:17,305 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:55:17,305 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:55:17,305 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:17,305 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:55:17,305 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:55:17,305 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:55:17,305 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. 2023-04-16 00:55:17,305 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. started... 2023-04-16 00:55:17,305 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. as already flushing 2023-04-16 00:55:17,305 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. 2023-04-16 00:55:17,305 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:55:17,305 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:55:17,305 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:17,306 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:55:17,306 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:55:17,307 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:17,307 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:17,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:17,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:17,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:17,307 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:17,307 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:55:17,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:17,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:17,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:17,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:17,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:17,308 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:55:17,308 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@22d5bec1[Count = 0] remaining members to acquire global barrier 2023-04-16 00:55:17,308 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:55:17,309 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:17,309 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:17,309 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:17,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:17,310 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:55:17,310 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:55:17,310 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:55:17,310 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:17,310 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:55:17,311 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:17,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:17,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:17,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:17,312 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:55:17,312 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:17,312 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:55:17,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:17,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:17,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:17,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:17,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:17,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:17,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:17,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:55:17,314 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:55:17,314 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:55:17,314 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:55:17,314 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:55:17,314 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:55:17,314 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:55:17,315 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:17,315 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:17,315 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:17,315 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:17,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:17,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:17,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:17,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:17,316 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:17,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:17,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:17,316 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:17,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:17,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:17,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:17,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:17,318 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:17,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:17,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:17,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:17,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:17,320 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:17,320 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:17,320 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:17,320 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:17,320 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:55:17,320 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:55:17,320 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:17,320 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:17,320 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:17,320 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:17,320 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:17,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:17,321 DEBUG [Thread-2414] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:55:17,321 DEBUG [Thread-2414] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:55:17,320 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:17,321 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:17,321 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:17,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:17,347 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230416d6f389e251784292816936f8a45e25d1 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230416d6f389e251784292816936f8a45e25d1 2023-04-16 00:55:17,421 DEBUG [Thread-2414] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:55:17,421 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:55:17,423 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:55:17,424 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:55:17,424 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:55:17,424 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:55:17,424 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:17,424 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:55:17,424 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:55:17,425 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:17,425 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:17,430 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:17,430 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:17,430 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:17,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:17,430 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:17,430 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:55:17,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:17,430 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:17,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:55:17,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:17,431 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:55:17,431 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:55:17,431 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:55:17,431 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:55:17,431 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:17,431 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:55:17,432 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:55:17,432 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:55:17,432 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. 2023-04-16 00:55:17,432 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. started... 2023-04-16 00:55:17,432 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. as already flushing 2023-04-16 00:55:17,432 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. 2023-04-16 00:55:17,432 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:55:17,432 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:55:17,432 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:17,432 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:55:17,432 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:55:17,433 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:17,433 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:17,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:17,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:17,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:17,434 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:17,434 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:55:17,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:17,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:17,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:17,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:17,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:17,436 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:55:17,436 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@12c2bdf2[Count = 0] remaining members to acquire global barrier 2023-04-16 00:55:17,436 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:55:17,436 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:17,437 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:17,437 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:17,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:17,437 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:55:17,437 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:17,437 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:55:17,437 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:55:17,438 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:55:17,439 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:55:17,439 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:17,439 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:17,439 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:55:17,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:17,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:17,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:17,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:17,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:17,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:17,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:17,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:17,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:17,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:17,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:55:17,441 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:55:17,441 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:55:17,441 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:55:17,441 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:55:17,441 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:55:17,441 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:55:17,444 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:17,444 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:17,444 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:17,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:17,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:17,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:17,444 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:17,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:17,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:17,445 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:17,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:17,445 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:17,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:17,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:17,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:17,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:17,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:17,446 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:17,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:17,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:17,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:17,448 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:17,448 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:17,448 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:17,448 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:17,448 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:17,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:17,448 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:55:17,448 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:17,448 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:17,448 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:17,448 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:17,448 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:55:17,448 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:55:17,448 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:17,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:17,448 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:17,448 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:17,448 DEBUG [Thread-2414] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:55:17,448 DEBUG [Thread-2414] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:55:17,538 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:17,539 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59642 deadline: 1681606577536, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:17,539 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:17,539 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59592 deadline: 1681606577536, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:17,539 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:17,539 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59666 deadline: 1681606577537, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:17,539 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:17,539 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59650 deadline: 1681606577538, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:17,542 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:17,543 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59664 deadline: 1681606577538, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:17,549 DEBUG [Thread-2414] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:55:17,549 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:55:17,551 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:55:17,551 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-16 00:55:17,552 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:55:17,552 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:55:17,552 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:55:17,552 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:17,552 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:55:17,552 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:55:17,553 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:17,553 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:17,554 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:17,554 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:17,554 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:17,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:17,554 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:17,554 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:55:17,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:17,555 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:17,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:55:17,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:17,555 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:55:17,555 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:55:17,555 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:55:17,555 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:55:17,555 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:17,555 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:55:17,556 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:55:17,556 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:55:17,556 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. 2023-04-16 00:55:17,556 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. started... 2023-04-16 00:55:17,556 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. as already flushing 2023-04-16 00:55:17,556 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. 2023-04-16 00:55:17,556 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:55:17,556 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:55:17,556 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:17,556 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:55:17,556 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:55:17,557 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:17,557 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:17,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:17,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:17,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:17,558 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:17,558 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:55:17,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:17,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:17,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:17,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:17,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:17,559 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:55:17,559 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:55:17,559 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3c88248[Count = 0] remaining members to acquire global barrier 2023-04-16 00:55:17,559 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:17,565 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:17,565 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:17,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:17,565 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:55:17,565 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:55:17,565 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:55:17,565 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:17,565 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:55:17,566 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:55:17,566 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:17,566 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:17,567 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:55:17,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:17,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:17,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:17,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:17,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:17,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:17,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:17,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:17,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:17,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:17,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:55:17,569 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:55:17,569 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:55:17,569 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:55:17,569 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:55:17,569 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:55:17,569 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:55:17,570 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:17,570 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:17,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:17,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:17,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:17,571 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:17,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:17,571 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:17,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:17,571 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:17,571 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:17,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:17,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:17,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:17,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:17,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:17,572 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:17,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:17,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:17,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:17,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:17,574 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:17,575 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:17,575 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:17,575 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:17,575 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:17,575 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:17,575 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:17,575 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:55:17,575 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:55:17,575 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:17,575 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:17,575 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:17,575 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:17,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:17,575 DEBUG [Thread-2414] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:55:17,575 DEBUG [Thread-2414] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:55:17,575 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:55:17,575 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:17,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:17,675 DEBUG [Thread-2414] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:55:17,676 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:55:17,677 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:55:17,678 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:55:17,678 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:55:17,678 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:55:17,679 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:17,679 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:55:17,679 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:55:17,679 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:17,679 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:17,682 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:17,682 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:17,682 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:17,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:17,682 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:17,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:17,682 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:55:17,682 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:17,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:55:17,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:17,683 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:55:17,683 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:55:17,683 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:55:17,683 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:55:17,683 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:17,683 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:55:17,683 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:55:17,683 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:55:17,683 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. 2023-04-16 00:55:17,683 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. started... 2023-04-16 00:55:17,683 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. as already flushing 2023-04-16 00:55:17,683 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. 2023-04-16 00:55:17,683 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:55:17,683 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:55:17,683 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:17,683 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:55:17,683 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:55:17,685 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:17,685 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:17,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:17,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:17,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:17,685 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:17,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:17,685 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:55:17,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:17,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:17,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:17,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:17,686 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:55:17,686 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@146e7dd1[Count = 0] remaining members to acquire global barrier 2023-04-16 00:55:17,686 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:55:17,687 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:17,687 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:17,687 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:17,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:17,688 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:55:17,688 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:55:17,688 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:55:17,688 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:17,688 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:55:17,689 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:17,689 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:55:17,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:17,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:17,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:17,689 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:17,689 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:55:17,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:17,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:17,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:17,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:17,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:17,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:17,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:17,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:55:17,691 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:55:17,691 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:55:17,691 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:55:17,691 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:55:17,691 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:55:17,691 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:55:17,693 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:17,693 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:17,693 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:17,693 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:17,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:17,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:17,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:17,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:17,694 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:17,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:17,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:17,695 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:17,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:17,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:17,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:17,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:17,696 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:17,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:17,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:17,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:17,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:17,697 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:17,697 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:17,697 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:17,697 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:17,697 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:17,697 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:17,697 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:55:17,698 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:55:17,698 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:17,698 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:17,698 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:17,698 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:17,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:17,698 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:17,698 DEBUG [Thread-2414] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:55:17,698 DEBUG [Thread-2414] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:55:17,698 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:17,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:17,752 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=15, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/.tmp/A/c16608668d934f83821faa9ccec59a9c 2023-04-16 00:55:17,764 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=15 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/.tmp/B/3ee45c1680e74c349af7a713cdd7d5bb 2023-04-16 00:55:17,798 DEBUG [Thread-2414] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:55:17,799 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:55:17,800 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:55:17,801 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:55:17,801 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:55:17,801 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:55:17,801 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:17,801 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:55:17,801 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:55:17,801 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:17,801 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:17,802 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:17,802 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:17,803 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:17,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:17,803 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:17,803 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:55:17,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:17,803 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:17,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:55:17,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:17,803 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:55:17,804 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:55:17,804 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:55:17,804 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:55:17,804 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:17,804 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:55:17,804 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:55:17,804 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:55:17,804 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. 2023-04-16 00:55:17,804 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. started... 2023-04-16 00:55:17,804 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. as already flushing 2023-04-16 00:55:17,804 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. 2023-04-16 00:55:17,804 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:55:17,804 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:55:17,804 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:17,804 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:55:17,804 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:55:17,812 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:17,812 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:17,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:17,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:17,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:17,812 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:17,812 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:55:17,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:17,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:17,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:17,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:17,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:17,813 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:55:17,813 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1b42fa2b[Count = 0] remaining members to acquire global barrier 2023-04-16 00:55:17,813 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:55:17,814 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:17,814 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:17,814 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:17,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:17,814 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:55:17,815 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:55:17,815 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:55:17,815 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:17,815 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:55:17,816 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:55:17,816 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:17,816 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:17,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:17,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:17,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:17,816 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:55:17,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:17,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:17,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:17,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:17,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:17,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:17,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:17,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:55:17,818 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:55:17,818 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:55:17,818 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:55:17,818 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:55:17,818 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:55:17,818 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:55:17,819 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:17,819 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:17,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:17,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:17,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:17,819 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:17,819 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:17,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:17,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:17,820 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:17,820 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:17,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:17,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:17,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:17,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:17,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:17,821 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:17,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:17,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:17,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:17,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:17,824 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:17,824 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:17,824 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:17,824 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:17,824 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:55:17,824 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:55:17,824 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:55:17,824 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:17,824 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:17,825 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:17,824 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:17,825 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:17,825 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:17,825 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:17,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:17,825 DEBUG [Thread-2414] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:55:17,825 DEBUG [Thread-2414] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:55:17,825 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:17,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:17,925 DEBUG [Thread-2414] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:55:17,926 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:55:17,927 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:55:17,928 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:55:17,928 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:55:17,928 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:55:17,928 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:17,928 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:55:17,928 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:55:17,929 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:17,929 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:17,930 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:17,930 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:17,930 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:17,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:17,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:17,930 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:17,930 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:55:17,931 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:17,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:55:17,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:17,931 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:55:17,931 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:55:17,932 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:55:17,932 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:55:17,932 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:17,932 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:55:17,932 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:55:17,932 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:55:17,932 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. 2023-04-16 00:55:17,932 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. started... 2023-04-16 00:55:17,932 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. as already flushing 2023-04-16 00:55:17,932 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. 2023-04-16 00:55:17,932 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:55:17,932 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:55:17,932 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:17,932 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:55:17,932 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:55:17,934 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:17,934 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:17,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:17,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:17,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:17,934 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:17,934 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:55:17,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:17,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:17,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:17,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:17,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:17,936 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:55:17,936 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1c68e99c[Count = 0] remaining members to acquire global barrier 2023-04-16 00:55:17,936 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:55:17,936 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:17,937 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:17,937 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:17,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:17,937 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:55:17,937 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:55:17,937 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:55:17,937 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:17,937 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:55:17,939 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:17,939 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:55:17,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:17,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:17,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:17,939 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:17,940 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:55:17,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:17,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:17,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:17,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:17,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:17,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:17,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:17,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:55:17,942 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:55:17,942 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:55:17,942 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:55:17,942 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:55:17,942 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:55:17,942 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:55:17,944 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:17,944 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:17,945 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:17,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:17,945 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:17,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:17,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:17,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:17,945 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:17,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:17,945 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:17,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:17,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:17,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:17,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:17,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:17,947 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:17,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:17,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:17,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:17,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:17,949 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:17,949 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:17,949 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:17,949 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:55:17,949 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:17,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:17,949 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:17,949 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:55:17,949 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:17,949 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:17,949 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:17,950 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:17,950 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:17,950 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:17,950 DEBUG [Thread-2414] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:55:17,950 DEBUG [Thread-2414] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:55:17,950 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:17,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:18,044 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:18,045 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59642 deadline: 1681606578042, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:18,045 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:18,045 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59666 deadline: 1681606578042, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:18,045 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:18,045 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59650 deadline: 1681606578042, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:18,050 DEBUG [Thread-2414] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:55:18,050 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:18,050 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59592 deadline: 1681606578044, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:18,051 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:18,051 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59664 deadline: 1681606578046, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:18,056 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:55:18,058 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:55:18,059 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:55:18,059 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:55:18,059 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:55:18,059 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:18,059 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:55:18,059 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:55:18,059 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:18,059 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:18,060 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:18,060 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:18,061 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:18,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:18,061 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:18,061 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:55:18,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:18,061 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:18,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:55:18,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:18,062 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:55:18,062 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:55:18,062 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:55:18,062 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:55:18,062 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:18,062 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:55:18,062 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:55:18,062 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:55:18,062 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. 2023-04-16 00:55:18,062 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. started... 2023-04-16 00:55:18,062 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. as already flushing 2023-04-16 00:55:18,063 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. 2023-04-16 00:55:18,063 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:55:18,063 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:55:18,063 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:18,063 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:55:18,063 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:55:18,066 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:18,066 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:18,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:18,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:18,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:18,066 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:18,066 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:55:18,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:18,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:18,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:18,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:18,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:18,067 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:55:18,067 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5b75e29b[Count = 0] remaining members to acquire global barrier 2023-04-16 00:55:18,067 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:55:18,068 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:18,069 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:18,069 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:18,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:18,069 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:55:18,069 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:18,069 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:55:18,069 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:55:18,069 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:55:18,070 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:55:18,070 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:18,070 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:18,070 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:55:18,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:18,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:18,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:18,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:18,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:18,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:18,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:18,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:18,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:18,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:18,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:55:18,072 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:55:18,073 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:55:18,073 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:55:18,073 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:55:18,073 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:55:18,073 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:55:18,074 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:18,074 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:18,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:18,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:18,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:18,074 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:18,074 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:18,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:18,074 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:18,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:18,074 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:18,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:18,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:18,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:18,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:18,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:18,076 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:18,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:18,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:18,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:18,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:18,080 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:18,080 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:18,080 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:18,080 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:55:18,080 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:55:18,080 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:18,080 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:18,080 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:18,080 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:18,080 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:18,080 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:18,080 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:18,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:18,080 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:18,080 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:55:18,080 DEBUG [Thread-2414] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:55:18,080 DEBUG [Thread-2414] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:55:18,080 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:18,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:18,174 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=15 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/.tmp/C/71299a44e639499e8a3034bde1608367 2023-04-16 00:55:18,178 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/.tmp/A/c16608668d934f83821faa9ccec59a9c as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/A/c16608668d934f83821faa9ccec59a9c 2023-04-16 00:55:18,180 DEBUG [Thread-2414] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:55:18,181 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:55:18,181 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/A/c16608668d934f83821faa9ccec59a9c, entries=150, sequenceid=15, filesize=25.1 K 2023-04-16 00:55:18,182 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/.tmp/B/3ee45c1680e74c349af7a713cdd7d5bb as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/B/3ee45c1680e74c349af7a713cdd7d5bb 2023-04-16 00:55:18,183 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:55:18,184 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:55:18,184 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:55:18,184 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:55:18,184 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:18,184 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:55:18,184 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:55:18,184 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:18,184 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:18,185 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:18,185 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:18,186 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:18,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:18,186 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/B/3ee45c1680e74c349af7a713cdd7d5bb, entries=150, sequenceid=15, filesize=11.6 K 2023-04-16 00:55:18,186 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:18,186 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:55:18,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:18,186 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/.tmp/C/71299a44e639499e8a3034bde1608367 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/C/71299a44e639499e8a3034bde1608367 2023-04-16 00:55:18,186 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:18,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:55:18,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:18,187 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:55:18,187 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:55:18,187 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:55:18,188 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:55:18,188 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:18,188 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:55:18,188 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:55:18,188 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:55:18,188 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. 2023-04-16 00:55:18,188 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. started... 2023-04-16 00:55:18,188 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. as already flushing 2023-04-16 00:55:18,188 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. 2023-04-16 00:55:18,188 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:55:18,188 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:55:18,188 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:18,188 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:55:18,188 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:55:18,189 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/C/71299a44e639499e8a3034bde1608367, entries=150, sequenceid=15, filesize=11.6 K 2023-04-16 00:55:18,190 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:18,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:18,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:18,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:18,190 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:18,190 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=134.18 KB/137400 for ee2f682ad6dd0fe8e91c021d54d3bdc5 in 1309ms, sequenceid=15, compaction requested=false 2023-04-16 00:55:18,190 DEBUG [MemStoreFlusher.0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'TestAcidGuarantees' 2023-04-16 00:55:18,191 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:18,191 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:55:18,191 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for ee2f682ad6dd0fe8e91c021d54d3bdc5: 2023-04-16 00:55:18,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:18,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:18,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:18,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:18,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:18,192 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:55:18,193 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3c129cd1[Count = 0] remaining members to acquire global barrier 2023-04-16 00:55:18,193 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:55:18,193 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:18,194 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:18,194 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:18,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:18,194 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:55:18,194 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:55:18,194 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:18,194 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:55:18,194 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:55:18,202 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:18,202 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:55:18,202 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:18,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:18,202 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:55:18,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:18,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:18,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:18,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:18,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:18,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:18,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:18,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:18,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:18,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:55:18,204 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:55:18,204 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:55:18,205 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:55:18,205 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:55:18,205 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:55:18,205 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:55:18,206 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:18,206 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:18,206 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:18,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:18,206 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:18,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:18,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:18,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:18,206 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:18,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:18,207 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:18,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:18,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:18,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:18,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:18,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:18,208 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:18,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:18,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:18,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:18,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:18,209 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:18,209 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:18,209 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:18,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:18,209 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:55:18,209 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:18,209 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:18,209 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:55:18,210 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:18,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:18,210 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:55:18,210 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:18,210 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:18,210 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:18,210 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:18,210 DEBUG [Thread-2414] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:55:18,210 DEBUG [Thread-2414] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:55:18,210 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:18,210 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:18,310 DEBUG [Thread-2414] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:55:18,311 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:55:18,312 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:55:18,313 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:55:18,313 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:55:18,313 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:55:18,314 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:18,314 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:55:18,314 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:55:18,314 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:18,314 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:18,315 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:18,315 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:18,315 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:18,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:18,315 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:18,315 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:55:18,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:18,316 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:18,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:55:18,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:18,316 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:55:18,316 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:55:18,317 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:55:18,317 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:55:18,317 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:18,317 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:55:18,317 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:55:18,317 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:55:18,317 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. 2023-04-16 00:55:18,317 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. started... 2023-04-16 00:55:18,317 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing ee2f682ad6dd0fe8e91c021d54d3bdc5 3/3 column families, dataSize=134.18 KB heapSize=361.55 KB 2023-04-16 00:55:18,317 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ee2f682ad6dd0fe8e91c021d54d3bdc5, store=A 2023-04-16 00:55:18,317 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 00:55:18,317 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ee2f682ad6dd0fe8e91c021d54d3bdc5, store=B 2023-04-16 00:55:18,317 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 00:55:18,318 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ee2f682ad6dd0fe8e91c021d54d3bdc5, store=C 2023-04-16 00:55:18,318 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 00:55:18,732 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304166f9c830f47b94e1ebf27375c688fd445 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304166f9c830f47b94e1ebf27375c688fd445 2023-04-16 00:55:18,738 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=47, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/.tmp/A/3674f578eab14439be5aa79c8c833cca 2023-04-16 00:55:18,750 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=47 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/.tmp/B/ce17b32b4d9743d79dbce98b90ba3f29 2023-04-16 00:55:19,054 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(9158): Flush requested on ee2f682ad6dd0fe8e91c021d54d3bdc5 2023-04-16 00:55:19,054 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. as already flushing 2023-04-16 00:55:19,063 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:19,063 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59592 deadline: 1681606579057, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:19,063 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:19,063 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59666 deadline: 1681606579058, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:19,068 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:19,068 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59650 deadline: 1681606579060, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:19,069 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:19,069 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59642 deadline: 1681606579061, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:19,069 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:19,070 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59664 deadline: 1681606579062, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:19,159 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=47 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/.tmp/C/fbdd963920664561b5cbe6b2544fc6c3 2023-04-16 00:55:19,163 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/.tmp/A/3674f578eab14439be5aa79c8c833cca as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/A/3674f578eab14439be5aa79c8c833cca 2023-04-16 00:55:19,167 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/A/3674f578eab14439be5aa79c8c833cca, entries=150, sequenceid=47, filesize=25.1 K 2023-04-16 00:55:19,167 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/.tmp/B/ce17b32b4d9743d79dbce98b90ba3f29 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/B/ce17b32b4d9743d79dbce98b90ba3f29 2023-04-16 00:55:19,169 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:19,169 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59592 deadline: 1681606579164, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:19,169 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:19,169 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59666 deadline: 1681606579164, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:19,172 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:19,172 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59650 deadline: 1681606579169, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:19,176 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:19,176 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59664 deadline: 1681606579172, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:19,176 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:19,176 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59642 deadline: 1681606579172, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:19,178 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/B/ce17b32b4d9743d79dbce98b90ba3f29, entries=150, sequenceid=47, filesize=11.6 K 2023-04-16 00:55:19,179 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/.tmp/C/fbdd963920664561b5cbe6b2544fc6c3 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/C/fbdd963920664561b5cbe6b2544fc6c3 2023-04-16 00:55:19,182 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/C/fbdd963920664561b5cbe6b2544fc6c3, entries=150, sequenceid=47, filesize=11.6 K 2023-04-16 00:55:19,183 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~361.50 KB/370176, currentSize=53.67 KB/54960 for ee2f682ad6dd0fe8e91c021d54d3bdc5 in 866ms, sequenceid=47, compaction requested=false 2023-04-16 00:55:19,184 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for ee2f682ad6dd0fe8e91c021d54d3bdc5: 2023-04-16 00:55:19,184 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. 2023-04-16 00:55:19,184 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:55:19,184 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:55:19,184 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:19,185 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:55:19,185 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:55:19,187 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:19,187 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:19,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:19,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:19,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:19,188 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:19,188 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:55:19,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:19,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:19,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:19,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:19,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:19,189 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:55:19,189 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@300dbada[Count = 0] remaining members to acquire global barrier 2023-04-16 00:55:19,189 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:55:19,189 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:19,190 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:19,190 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:19,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:19,191 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:55:19,191 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:55:19,191 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:19,191 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:55:19,191 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:55:19,193 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:19,193 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:55:19,193 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:19,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:19,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:19,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:19,193 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:55:19,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:19,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:19,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:19,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:19,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:19,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:19,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:19,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:55:19,195 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:55:19,195 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:55:19,195 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:55:19,195 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:55:19,195 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:55:19,195 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:55:19,197 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:19,197 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:19,197 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:19,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:19,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:19,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:19,197 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:19,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:19,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:19,198 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:19,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:19,198 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:19,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:19,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:19,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:19,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:19,210 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:19,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:19,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:19,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:19,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:19,213 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:19,214 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:19,214 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:19,214 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:19,214 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:19,214 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:19,214 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:19,214 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:19,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:19,214 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:19,214 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:19,214 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:55:19,214 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:55:19,214 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:19,214 DEBUG [Thread-2414] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:55:19,214 DEBUG [Thread-2414] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:55:19,214 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:19,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:19,215 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:55:19,314 DEBUG [Thread-2414] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:55:19,315 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:55:19,317 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:55:19,320 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:55:19,320 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:55:19,320 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:55:19,320 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:19,320 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:55:19,320 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:55:19,320 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:19,321 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:19,322 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:19,322 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:19,322 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:19,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:19,322 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:19,323 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:55:19,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:19,323 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:19,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:55:19,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:19,323 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:55:19,326 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:55:19,326 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:55:19,326 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:55:19,326 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:19,326 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:55:19,326 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:55:19,326 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:55:19,327 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. 2023-04-16 00:55:19,327 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. started... 2023-04-16 00:55:19,327 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing ee2f682ad6dd0fe8e91c021d54d3bdc5 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-16 00:55:19,327 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ee2f682ad6dd0fe8e91c021d54d3bdc5, store=A 2023-04-16 00:55:19,327 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:55:19,327 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ee2f682ad6dd0fe8e91c021d54d3bdc5, store=B 2023-04-16 00:55:19,327 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:55:19,327 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ee2f682ad6dd0fe8e91c021d54d3bdc5, store=C 2023-04-16 00:55:19,327 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:55:19,376 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(9158): Flush requested on ee2f682ad6dd0fe8e91c021d54d3bdc5 2023-04-16 00:55:19,376 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. as already flushing 2023-04-16 00:55:19,401 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1681606519385, max timestamp=1681606519399Num uniques 859; 2023-04-16 00:55:19,401 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-16 00:55:19,401 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1681606519385, max timestamp=1681606519399Num uniques 859; 2023-04-16 00:55:19,401 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-16 00:55:19,401 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1681606519385, max timestamp=1681606519399Num uniques 859; 2023-04-16 00:55:19,401 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-16 00:55:19,427 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:19,427 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59650 deadline: 1681606579418, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:19,427 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:19,427 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59666 deadline: 1681606579419, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:19,428 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:19,428 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59664 deadline: 1681606579420, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:19,435 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:19,435 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59592 deadline: 1681606579427, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:19,435 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:19,435 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59642 deadline: 1681606579427, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:19,533 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:19,533 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59650 deadline: 1681606579528, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:19,533 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:19,533 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59666 deadline: 1681606579528, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:19,533 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:19,534 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59664 deadline: 1681606579528, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:19,541 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:19,541 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59592 deadline: 1681606579536, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:19,541 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:19,541 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59642 deadline: 1681606579536, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:19,737 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:19,737 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59666 deadline: 1681606579734, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:19,737 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:19,737 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59650 deadline: 1681606579734, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:19,740 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:19,741 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59664 deadline: 1681606579734, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:19,745 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:19,745 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59642 deadline: 1681606579742, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:19,747 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:19,747 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59592 deadline: 1681606579743, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:19,777 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023041677904b95d71b4d72b4cd7fbfd3678e80 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041677904b95d71b4d72b4cd7fbfd3678e80 2023-04-16 00:55:20,041 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:20,042 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59666 deadline: 1681606580038, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:20,042 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:20,042 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59650 deadline: 1681606580039, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:20,049 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:20,050 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59664 deadline: 1681606580043, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:20,050 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:20,050 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59642 deadline: 1681606580046, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:20,053 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:20,053 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59592 deadline: 1681606580049, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:20,182 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=58, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/.tmp/A/a75b8168cc3348629249af945ef659d6 2023-04-16 00:55:20,553 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:20,553 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59666 deadline: 1681606580545, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:20,554 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:20,554 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59650 deadline: 1681606580545, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:20,555 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:20,555 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59664 deadline: 1681606580550, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:20,562 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:20,562 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59642 deadline: 1681606580555, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:20,562 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:20,563 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59592 deadline: 1681606580556, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:20,592 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=58 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/.tmp/B/fe881913baab42d4ac296653925cddba 2023-04-16 00:55:21,006 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=58 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/.tmp/C/dee497a3a72947de84df13174cb2a796 2023-04-16 00:55:21,010 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/.tmp/A/a75b8168cc3348629249af945ef659d6 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/A/a75b8168cc3348629249af945ef659d6 2023-04-16 00:55:21,013 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/A/a75b8168cc3348629249af945ef659d6, entries=150, sequenceid=58, filesize=25.1 K 2023-04-16 00:55:21,014 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/.tmp/B/fe881913baab42d4ac296653925cddba as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/B/fe881913baab42d4ac296653925cddba 2023-04-16 00:55:21,017 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/B/fe881913baab42d4ac296653925cddba, entries=150, sequenceid=58, filesize=11.6 K 2023-04-16 00:55:21,022 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/.tmp/C/dee497a3a72947de84df13174cb2a796 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/C/dee497a3a72947de84df13174cb2a796 2023-04-16 00:55:21,026 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/C/dee497a3a72947de84df13174cb2a796, entries=150, sequenceid=58, filesize=11.6 K 2023-04-16 00:55:21,027 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=140.89 KB/144270 for ee2f682ad6dd0fe8e91c021d54d3bdc5 in 1700ms, sequenceid=58, compaction requested=true 2023-04-16 00:55:21,027 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for ee2f682ad6dd0fe8e91c021d54d3bdc5: 2023-04-16 00:55:21,027 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. 2023-04-16 00:55:21,027 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:55:21,027 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:55:21,027 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:21,027 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:55:21,027 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:55:21,031 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:21,031 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:21,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:21,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:21,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:21,031 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:21,031 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:55:21,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:21,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:21,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:21,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:21,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:21,032 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:55:21,033 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6bf2914e[Count = 0] remaining members to acquire global barrier 2023-04-16 00:55:21,033 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:55:21,033 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:21,034 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:21,034 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:21,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:21,034 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:55:21,034 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:55:21,034 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:55:21,034 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:21,034 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:55:21,037 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:21,037 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:55:21,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:21,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:21,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:21,037 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:21,038 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:55:21,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:21,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:21,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:21,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:21,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:21,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:21,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:21,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:55:21,040 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:55:21,040 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:55:21,040 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:55:21,040 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:55:21,040 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:55:21,040 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:55:21,042 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:21,042 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:21,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:21,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:21,042 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:21,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:21,042 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:21,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:21,042 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:21,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:21,042 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:21,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:21,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:21,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:21,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:21,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:21,043 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:21,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:21,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:21,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:21,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:21,046 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:21,046 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:21,046 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:21,046 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:21,046 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:21,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:21,046 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:55:21,046 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:55:21,046 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:55:21,046 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:21,046 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:21,046 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:21,046 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:21,046 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:21,046 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:21,046 DEBUG [Thread-2414] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:55:21,046 DEBUG [Thread-2414] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:55:21,046 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:21,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:21,146 DEBUG [Thread-2414] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:55:21,147 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:55:21,148 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:55:21,149 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:55:21,149 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:55:21,149 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:55:21,150 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:21,150 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:55:21,150 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:55:21,150 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:21,150 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:21,151 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:21,151 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:21,151 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:21,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:21,151 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:21,152 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:55:21,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:21,152 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:21,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:55:21,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:21,152 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:55:21,153 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:55:21,153 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:55:21,153 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:55:21,153 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:21,153 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:55:21,153 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:55:21,153 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:55:21,153 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. 2023-04-16 00:55:21,153 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. started... 2023-04-16 00:55:21,153 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing ee2f682ad6dd0fe8e91c021d54d3bdc5 3/3 column families, dataSize=140.89 KB heapSize=390.84 KB 2023-04-16 00:55:21,153 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ee2f682ad6dd0fe8e91c021d54d3bdc5, store=A 2023-04-16 00:55:21,153 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 00:55:21,154 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ee2f682ad6dd0fe8e91c021d54d3bdc5, store=B 2023-04-16 00:55:21,154 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 00:55:21,154 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ee2f682ad6dd0fe8e91c021d54d3bdc5, store=C 2023-04-16 00:55:21,154 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 00:55:21,559 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(9158): Flush requested on ee2f682ad6dd0fe8e91c021d54d3bdc5 2023-04-16 00:55:21,559 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. as already flushing 2023-04-16 00:55:21,582 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230416c00629f1e82e49db975e698aa16e4542 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230416c00629f1e82e49db975e698aa16e4542 2023-04-16 00:55:21,585 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:21,585 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59666 deadline: 1681606581570, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:21,585 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:21,585 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59664 deadline: 1681606581570, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:21,585 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:21,585 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59642 deadline: 1681606581571, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:21,585 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:21,585 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59592 deadline: 1681606581571, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:21,586 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:21,586 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59650 deadline: 1681606581572, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:21,590 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=93, memsize=47.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/.tmp/A/37f399b95f1b49f5af8d5a410b18d60f 2023-04-16 00:55:21,601 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=93 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/.tmp/B/c9a7176dd8534721976f4dbc1076206d 2023-04-16 00:55:21,615 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=93 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/.tmp/C/9e1ac9a463654bab82cbfd048ba6294f 2023-04-16 00:55:21,619 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/.tmp/A/37f399b95f1b49f5af8d5a410b18d60f as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/A/37f399b95f1b49f5af8d5a410b18d60f 2023-04-16 00:55:21,623 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/A/37f399b95f1b49f5af8d5a410b18d60f, entries=150, sequenceid=93, filesize=25.1 K 2023-04-16 00:55:21,624 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/.tmp/B/c9a7176dd8534721976f4dbc1076206d as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/B/c9a7176dd8534721976f4dbc1076206d 2023-04-16 00:55:21,631 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/B/c9a7176dd8534721976f4dbc1076206d, entries=150, sequenceid=93, filesize=11.6 K 2023-04-16 00:55:21,631 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/.tmp/C/9e1ac9a463654bab82cbfd048ba6294f as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/C/9e1ac9a463654bab82cbfd048ba6294f 2023-04-16 00:55:21,648 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/C/9e1ac9a463654bab82cbfd048ba6294f, entries=150, sequenceid=93, filesize=11.6 K 2023-04-16 00:55:21,650 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~390.80 KB/400176, currentSize=46.96 KB/48090 for ee2f682ad6dd0fe8e91c021d54d3bdc5 in 497ms, sequenceid=93, compaction requested=true 2023-04-16 00:55:21,651 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for ee2f682ad6dd0fe8e91c021d54d3bdc5: 2023-04-16 00:55:21,651 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. 2023-04-16 00:55:21,651 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:55:21,651 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:55:21,651 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:21,651 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:55:21,651 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:55:21,655 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:21,655 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:21,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:21,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:21,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:21,656 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:21,656 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:55:21,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:21,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:21,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:21,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:21,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:21,657 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:55:21,657 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1f67d791[Count = 0] remaining members to acquire global barrier 2023-04-16 00:55:21,657 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:55:21,657 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:21,658 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:21,658 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:21,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:21,659 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:55:21,659 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:55:21,659 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:55:21,659 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:21,659 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:55:21,660 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:55:21,660 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:21,660 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:21,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:21,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:21,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:21,660 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:55:21,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:21,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:21,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:21,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:21,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:21,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:21,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:21,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:55:21,668 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:55:21,668 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:55:21,669 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:55:21,669 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:55:21,669 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:55:21,669 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:55:21,673 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:21,673 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:21,673 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:21,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:21,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:21,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:21,673 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:21,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:21,674 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:21,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:21,675 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:21,675 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:21,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:21,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:21,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:21,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:21,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:21,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:21,680 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:21,680 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:21,680 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:21,680 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:21,680 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:21,680 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:21,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:21,680 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:21,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:21,680 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:21,680 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:55:21,680 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:55:21,680 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:21,680 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:21,681 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:21,681 DEBUG [Thread-2414] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:55:21,681 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:21,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:21,681 DEBUG [Thread-2414] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:55:21,681 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:55:21,697 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(9158): Flush requested on ee2f682ad6dd0fe8e91c021d54d3bdc5 2023-04-16 00:55:21,697 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing ee2f682ad6dd0fe8e91c021d54d3bdc5 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-16 00:55:21,698 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ee2f682ad6dd0fe8e91c021d54d3bdc5, store=A 2023-04-16 00:55:21,698 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:55:21,698 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ee2f682ad6dd0fe8e91c021d54d3bdc5, store=B 2023-04-16 00:55:21,698 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:55:21,698 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ee2f682ad6dd0fe8e91c021d54d3bdc5, store=C 2023-04-16 00:55:21,698 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:55:21,745 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681606521736, max timestamp=1681606521743Num uniques 859; 2023-04-16 00:55:21,746 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-16 00:55:21,746 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681606521736, max timestamp=1681606521743Num uniques 859; 2023-04-16 00:55:21,746 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-16 00:55:21,746 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681606521736, max timestamp=1681606521743Num uniques 859; 2023-04-16 00:55:21,746 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-16 00:55:21,779 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:21,779 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59642 deadline: 1681606581771, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:21,779 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:21,779 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59650 deadline: 1681606581771, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:21,781 DEBUG [Thread-2414] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:55:21,781 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:55:21,783 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:55:21,784 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:55:21,784 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:55:21,784 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:55:21,784 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:21,784 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:55:21,784 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:55:21,784 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:21,784 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:21,785 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:21,785 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:21,785 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:21,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:21,786 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:21,786 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59666 deadline: 1681606581778, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:21,786 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:21,786 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:55:21,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:21,792 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:21,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:55:21,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:21,793 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:55:21,793 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:55:21,793 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:55:21,793 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:21,793 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:55:21,793 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59664 deadline: 1681606581779, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:21,793 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:21,793 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:55:21,796 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:55:21,796 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:55:21,796 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. 2023-04-16 00:55:21,796 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. started... 2023-04-16 00:55:21,796 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. as already flushing 2023-04-16 00:55:21,796 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. 2023-04-16 00:55:21,797 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:55:21,797 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:55:21,797 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:21,797 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:55:21,797 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:55:21,799 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:21,799 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:21,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:21,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:21,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:21,799 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:21,799 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:55:21,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:21,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:21,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:21,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:21,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:21,800 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:55:21,801 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@37bb34b8[Count = 0] remaining members to acquire global barrier 2023-04-16 00:55:21,801 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:55:21,801 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:21,802 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:21,802 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:21,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:21,802 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:55:21,802 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:55:21,802 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:55:21,802 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:21,802 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:55:21,803 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:21,803 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:55:21,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:21,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:21,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:21,803 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:21,803 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:55:21,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:21,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:21,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:21,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:21,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:21,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:21,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:21,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:55:21,805 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:55:21,805 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:55:21,805 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:55:21,805 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:55:21,805 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:55:21,805 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:55:21,807 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:21,807 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:21,807 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:21,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:21,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:21,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:21,807 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:21,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:21,807 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:21,807 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:21,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:21,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:21,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:21,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:21,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:21,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:21,810 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:21,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:21,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:21,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:21,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:21,812 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:21,812 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:21,812 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:21,812 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:55:21,812 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:55:21,812 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:21,812 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:21,812 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:55:21,812 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:21,812 DEBUG [Thread-2414] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:55:21,813 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:21,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:21,813 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:21,813 DEBUG [Thread-2414] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:55:21,813 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:21,813 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:21,813 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:21,813 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:21,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:21,886 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:21,886 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59642 deadline: 1681606581880, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:21,886 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:21,886 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59650 deadline: 1681606581880, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:21,892 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:21,892 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59666 deadline: 1681606581887, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:21,899 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:21,899 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59664 deadline: 1681606581894, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:21,913 DEBUG [Thread-2414] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:55:21,913 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:55:21,914 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:55:21,916 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:55:21,916 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:55:21,916 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:55:21,916 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:21,916 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:55:21,916 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:55:21,916 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:21,916 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:21,917 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:21,918 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:21,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:21,918 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:21,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:21,918 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:21,918 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:55:21,918 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:21,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:55:21,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:21,918 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:55:21,919 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:55:21,919 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:55:21,919 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:55:21,919 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:21,919 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:55:21,919 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:55:21,919 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:55:21,919 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. 2023-04-16 00:55:21,919 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. started... 2023-04-16 00:55:21,919 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. as already flushing 2023-04-16 00:55:21,919 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. 2023-04-16 00:55:21,919 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:55:21,919 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:55:21,919 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:21,919 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:55:21,919 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:55:21,921 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:21,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:21,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:21,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:21,921 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:21,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:21,921 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:21,921 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:55:21,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:21,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:21,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:21,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:21,922 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:55:21,922 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:55:21,922 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6a3d83ff[Count = 0] remaining members to acquire global barrier 2023-04-16 00:55:21,922 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:21,923 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:21,923 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:21,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:21,923 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:55:21,923 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:55:21,923 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:55:21,924 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:21,924 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:55:21,926 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:55:21,926 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:21,926 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:21,926 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:55:21,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:21,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:21,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:21,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:21,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:21,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:21,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:21,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:21,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:21,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:21,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:55:21,929 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:55:21,929 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:55:21,929 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:55:21,929 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:55:21,929 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:55:21,929 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:55:21,931 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:21,931 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:21,931 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:21,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:21,931 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:21,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:21,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:21,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:21,932 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:21,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:21,932 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:21,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:21,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:21,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:21,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:21,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:21,933 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:21,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:21,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:21,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:21,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:21,935 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:21,935 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:21,935 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:21,935 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:21,935 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:21,935 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:55:21,935 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:21,935 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:55:21,935 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:21,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:21,935 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:21,935 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:55:21,935 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:21,935 DEBUG [Thread-2414] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:55:21,935 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:21,935 DEBUG [Thread-2414] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:55:21,935 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:21,935 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:21,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:22,035 DEBUG [Thread-2414] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:55:22,036 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:55:22,038 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:55:22,039 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:55:22,039 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:55:22,039 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:55:22,039 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:22,039 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:55:22,039 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:55:22,039 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:22,039 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:22,040 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:22,040 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:22,040 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:22,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:22,041 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:22,041 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:55:22,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:22,041 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:22,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:55:22,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:22,041 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:55:22,042 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:55:22,042 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:55:22,042 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:55:22,042 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:22,042 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:55:22,043 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:55:22,043 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:55:22,043 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. 2023-04-16 00:55:22,043 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. started... 2023-04-16 00:55:22,043 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. as already flushing 2023-04-16 00:55:22,043 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. 2023-04-16 00:55:22,043 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:55:22,043 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:55:22,043 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:22,043 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:55:22,043 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:55:22,045 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:22,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:22,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:22,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:22,046 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:22,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:22,046 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:22,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:22,046 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:55:22,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:22,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:22,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:22,047 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:55:22,047 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:55:22,047 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@153a1717[Count = 0] remaining members to acquire global barrier 2023-04-16 00:55:22,047 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:22,048 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:22,048 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:22,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:22,049 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:55:22,049 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:22,049 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:55:22,049 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:55:22,049 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:55:22,050 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:22,050 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:55:22,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:22,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:22,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:22,050 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:22,050 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:55:22,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:22,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:22,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:22,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:22,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:22,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:22,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:22,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:55:22,052 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:55:22,052 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:55:22,052 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:55:22,052 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:55:22,052 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:55:22,052 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:55:22,053 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:22,053 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:22,053 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:22,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:22,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:22,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:22,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:22,053 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:22,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:22,054 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:22,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:22,054 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:22,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:22,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:22,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:22,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:22,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:22,055 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:22,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:22,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:22,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:22,058 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:22,058 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:22,058 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:22,058 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:22,058 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:22,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:22,058 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:22,058 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:55:22,058 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:22,058 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:55:22,058 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:55:22,058 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:22,058 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:22,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:22,058 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:22,059 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:22,059 DEBUG [Thread-2414] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:55:22,059 DEBUG [Thread-2414] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:55:22,059 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:22,097 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:22,097 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59650 deadline: 1681606582089, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:22,097 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:22,097 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59642 deadline: 1681606582089, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:22,098 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:22,098 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59666 deadline: 1681606582093, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:22,108 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:22,108 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59664 deadline: 1681606582100, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:22,116 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023041695f2c39d277243af86e1f816526a4ef0 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041695f2c39d277243af86e1f816526a4ef0 2023-04-16 00:55:22,126 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=104, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/.tmp/A/2ed4b45335904963b9fb9d185fb956b0 2023-04-16 00:55:22,159 DEBUG [Thread-2414] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:55:22,159 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:55:22,162 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:55:22,176 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:55:22,176 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:55:22,176 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:55:22,176 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:22,176 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:55:22,176 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:55:22,177 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:22,177 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:22,179 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:22,179 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:22,179 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:22,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:22,180 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:22,180 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:55:22,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:22,180 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:22,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:55:22,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:22,181 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:55:22,189 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:55:22,189 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:55:22,189 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:22,190 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:55:22,190 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:55:22,190 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:55:22,190 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. 2023-04-16 00:55:22,190 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. started... 2023-04-16 00:55:22,190 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. as already flushing 2023-04-16 00:55:22,190 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. 2023-04-16 00:55:22,190 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:55:22,190 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:55:22,190 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:22,190 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:55:22,190 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:55:22,192 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:22,192 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:22,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:22,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:22,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:22,193 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:22,193 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:55:22,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:22,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:22,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:22,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:22,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:22,194 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:55:22,194 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@34c0917e[Count = 0] remaining members to acquire global barrier 2023-04-16 00:55:22,194 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:55:22,194 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:22,196 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:22,196 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:22,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:22,196 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:55:22,196 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:22,196 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:55:22,196 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:55:22,196 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:55:22,200 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:22,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:22,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:22,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:22,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:22,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:22,201 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:55:22,201 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:22,201 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:55:22,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:22,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:22,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:22,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:22,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:22,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:55:22,203 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:55:22,203 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:55:22,203 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:55:22,203 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:55:22,203 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:55:22,203 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:55:22,205 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:22,205 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:22,205 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:22,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:22,206 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:22,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:22,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:22,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:22,206 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:22,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:22,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:22,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:22,207 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:22,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:22,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:22,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:22,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:22,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:22,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:22,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:22,210 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:22,214 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:22,214 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:22,214 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:22,214 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:22,214 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:22,214 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:22,214 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:22,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:22,214 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:22,214 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:55:22,214 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:55:22,214 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:22,214 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:22,214 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:22,215 DEBUG [Thread-2414] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:55:22,215 DEBUG [Thread-2414] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:55:22,215 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:22,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:22,315 DEBUG [Thread-2414] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:55:22,315 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:55:22,317 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:55:22,318 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:55:22,318 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:55:22,318 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:55:22,318 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:22,318 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:55:22,318 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:55:22,318 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:22,318 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:22,320 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:22,320 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:22,320 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:22,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:22,320 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:22,320 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:55:22,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:22,320 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:22,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:55:22,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:22,321 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:55:22,321 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:55:22,321 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:55:22,321 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:55:22,321 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:22,321 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:55:22,321 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:55:22,321 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:55:22,321 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. 2023-04-16 00:55:22,322 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. started... 2023-04-16 00:55:22,322 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. as already flushing 2023-04-16 00:55:22,322 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. 2023-04-16 00:55:22,322 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:55:22,322 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:55:22,322 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:22,322 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:55:22,322 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:55:22,324 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:22,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:22,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:22,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:22,324 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:22,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:22,325 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:22,325 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:55:22,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:22,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:22,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:22,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:22,326 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:55:22,326 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5a307daa[Count = 0] remaining members to acquire global barrier 2023-04-16 00:55:22,326 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:55:22,326 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:22,327 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:22,327 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:22,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:22,327 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:55:22,327 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:55:22,327 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:55:22,327 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:22,327 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:55:22,329 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:22,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:22,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:22,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:22,329 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:55:22,329 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:22,329 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:55:22,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:22,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:22,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:22,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:22,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:22,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:22,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:22,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:55:22,331 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:55:22,331 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:55:22,331 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:55:22,331 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:55:22,331 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:55:22,331 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:55:22,332 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:22,332 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:22,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:22,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:22,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:22,333 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:22,333 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:22,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:22,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:22,333 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:22,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:22,333 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:22,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:22,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:22,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:22,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:22,334 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:22,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:22,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:22,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:22,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:22,337 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:22,337 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:22,337 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:22,338 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:55:22,338 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:55:22,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:22,338 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:22,338 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:22,338 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:22,338 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:22,338 DEBUG [Thread-2414] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:55:22,338 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:22,338 DEBUG [Thread-2414] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:55:22,338 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:22,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:22,338 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:22,338 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:22,338 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:22,406 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:22,407 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59642 deadline: 1681606582399, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:22,407 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:22,407 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59650 deadline: 1681606582399, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:22,410 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:22,410 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59666 deadline: 1681606582401, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:22,418 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:22,418 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59664 deadline: 1681606582410, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:22,438 DEBUG [Thread-2414] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:55:22,439 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:55:22,440 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:55:22,441 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:55:22,441 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:55:22,441 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:55:22,441 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:22,442 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:55:22,442 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:55:22,442 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:22,442 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:22,443 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:22,443 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:22,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:22,443 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:22,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:22,444 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:22,444 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:55:22,444 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:22,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:55:22,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:22,445 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:55:22,445 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:55:22,445 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:55:22,445 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:55:22,445 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:22,445 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:55:22,445 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:55:22,445 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:55:22,447 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. 2023-04-16 00:55:22,447 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. started... 2023-04-16 00:55:22,447 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. as already flushing 2023-04-16 00:55:22,447 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. 2023-04-16 00:55:22,447 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:55:22,447 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:55:22,448 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:22,448 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:55:22,448 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:55:22,449 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:22,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:22,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:22,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:22,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:22,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:22,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:22,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:22,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:22,449 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:22,451 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:55:22,451 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5c943f68[Count = 0] remaining members to acquire global barrier 2023-04-16 00:55:22,451 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:22,451 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:55:22,451 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:22,451 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:55:22,452 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:22,452 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:22,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:22,452 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:55:22,452 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:55:22,452 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:55:22,453 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:22,453 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:55:22,455 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:55:22,455 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:22,455 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:22,455 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:55:22,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:22,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:22,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:22,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:22,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:22,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:22,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:22,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:22,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:22,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:22,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:55:22,459 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:55:22,459 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:55:22,459 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:55:22,460 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:55:22,460 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:55:22,460 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:55:22,462 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:22,463 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:22,463 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:22,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:22,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:22,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:22,463 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:22,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:22,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:22,463 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:22,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:22,464 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:22,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:22,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:22,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:22,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:22,465 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:22,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:22,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:22,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:22,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:22,467 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:22,467 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:22,467 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:22,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:22,468 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:22,468 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:22,468 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:22,468 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:22,468 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:22,468 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:22,468 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:22,468 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:22,468 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:55:22,468 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:55:22,468 DEBUG [Thread-2414] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:55:22,468 DEBUG [Thread-2414] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:55:22,468 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:22,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:22,569 DEBUG [Thread-2414] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:55:22,569 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:55:22,571 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:55:22,572 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:55:22,572 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:55:22,572 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:55:22,572 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:22,572 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:55:22,572 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:55:22,574 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=104 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/.tmp/B/b73a63612e3049c4b22bb00e99791539 2023-04-16 00:55:22,574 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:22,574 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:22,578 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:22,579 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:22,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:22,579 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:22,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:22,579 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:22,579 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:22,580 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:55:22,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:55:22,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:22,580 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:55:22,581 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:55:22,581 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:55:22,581 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:55:22,581 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:22,581 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:55:22,584 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:55:22,584 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:55:22,584 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. 2023-04-16 00:55:22,584 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. started... 2023-04-16 00:55:22,585 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. as already flushing 2023-04-16 00:55:22,585 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. 2023-04-16 00:55:22,585 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:55:22,585 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:55:22,585 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:22,585 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:55:22,585 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:55:22,587 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:22,587 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:22,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:22,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:22,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:22,587 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:22,587 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:55:22,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:22,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:22,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:22,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:22,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:22,591 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:55:22,591 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@19d31942[Count = 0] remaining members to acquire global barrier 2023-04-16 00:55:22,591 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:55:22,591 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:22,592 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:22,592 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:22,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:22,593 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:22,593 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:55:22,595 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:55:22,595 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=104 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/.tmp/C/ada9cb30cf754d93858849098089397e 2023-04-16 00:55:22,595 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:55:22,595 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:55:22,598 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:55:22,598 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:22,598 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:55:22,598 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:22,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:22,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:22,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:22,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:22,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:22,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:22,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:22,599 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/.tmp/A/2ed4b45335904963b9fb9d185fb956b0 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/A/2ed4b45335904963b9fb9d185fb956b0 2023-04-16 00:55:22,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:22,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:22,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:22,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:55:22,601 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:55:22,601 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:55:22,601 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:55:22,601 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:55:22,601 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:55:22,601 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:55:22,603 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/A/2ed4b45335904963b9fb9d185fb956b0, entries=150, sequenceid=104, filesize=25.1 K 2023-04-16 00:55:22,604 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:22,604 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:22,604 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:22,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:22,604 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:22,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:22,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:22,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:22,604 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:22,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:22,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:22,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:22,605 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/.tmp/B/b73a63612e3049c4b22bb00e99791539 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/B/b73a63612e3049c4b22bb00e99791539 2023-04-16 00:55:22,605 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:22,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:22,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:22,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:22,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:22,607 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:22,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:22,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:22,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:22,611 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:22,611 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:22,611 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:22,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:22,611 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:22,611 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:22,611 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:22,611 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:22,611 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:22,611 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:22,611 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:22,611 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:22,611 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:22,611 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:55:22,611 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:55:22,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:22,612 DEBUG [Thread-2414] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:55:22,612 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/B/b73a63612e3049c4b22bb00e99791539, entries=150, sequenceid=104, filesize=11.6 K 2023-04-16 00:55:22,612 DEBUG [Thread-2414] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:55:22,613 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/.tmp/C/ada9cb30cf754d93858849098089397e as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/C/ada9cb30cf754d93858849098089397e 2023-04-16 00:55:22,616 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/C/ada9cb30cf754d93858849098089397e, entries=150, sequenceid=104, filesize=11.6 K 2023-04-16 00:55:22,617 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for ee2f682ad6dd0fe8e91c021d54d3bdc5 in 920ms, sequenceid=104, compaction requested=true 2023-04-16 00:55:22,617 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for ee2f682ad6dd0fe8e91c021d54d3bdc5: 2023-04-16 00:55:22,617 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 00:55:22,617 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-16 00:55:22,617 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-16 00:55:22,617 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 00:55:22,618 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 00:55:22,619 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 59430 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-16 00:55:22,619 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HStore(1912): ee2f682ad6dd0fe8e91c021d54d3bdc5/B is initiating minor compaction (all files) 2023-04-16 00:55:22,619 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HRegion(2259): Starting compaction of ee2f682ad6dd0fe8e91c021d54d3bdc5/B in TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. 2023-04-16 00:55:22,619 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 128730 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-16 00:55:22,619 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/B/3ee45c1680e74c349af7a713cdd7d5bb, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/B/ce17b32b4d9743d79dbce98b90ba3f29, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/B/fe881913baab42d4ac296653925cddba, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/B/c9a7176dd8534721976f4dbc1076206d, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/B/b73a63612e3049c4b22bb00e99791539] into tmpdir=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/.tmp, totalSize=58.0 K 2023-04-16 00:55:22,619 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HStore(1912): ee2f682ad6dd0fe8e91c021d54d3bdc5/A is initiating minor compaction (all files) 2023-04-16 00:55:22,620 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of ee2f682ad6dd0fe8e91c021d54d3bdc5/A in TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. 2023-04-16 00:55:22,620 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/A/c16608668d934f83821faa9ccec59a9c, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/A/3674f578eab14439be5aa79c8c833cca, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/A/a75b8168cc3348629249af945ef659d6, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/A/37f399b95f1b49f5af8d5a410b18d60f, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/A/2ed4b45335904963b9fb9d185fb956b0] into tmpdir=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/.tmp, totalSize=125.7 K 2023-04-16 00:55:22,620 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting 3ee45c1680e74c349af7a713cdd7d5bb, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=15, earliestPutTs=1681606516876 2023-04-16 00:55:22,620 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting ce17b32b4d9743d79dbce98b90ba3f29, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=47, earliestPutTs=1681606516912 2023-04-16 00:55:22,621 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.Compactor(207): Compacting c16608668d934f83821faa9ccec59a9c, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=15, earliestPutTs=1681606516876 2023-04-16 00:55:22,621 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting fe881913baab42d4ac296653925cddba, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=58, earliestPutTs=1681606519057 2023-04-16 00:55:22,622 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.Compactor(207): Compacting 3674f578eab14439be5aa79c8c833cca, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=47, earliestPutTs=1681606516912 2023-04-16 00:55:22,622 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.Compactor(207): Compacting a75b8168cc3348629249af945ef659d6, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=58, earliestPutTs=1681606519057 2023-04-16 00:55:22,622 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting c9a7176dd8534721976f4dbc1076206d, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=93, earliestPutTs=1681606519417 2023-04-16 00:55:22,623 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.Compactor(207): Compacting 37f399b95f1b49f5af8d5a410b18d60f, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=93, earliestPutTs=1681606519417 2023-04-16 00:55:22,623 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting b73a63612e3049c4b22bb00e99791539, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=104, earliestPutTs=1681606521569 2023-04-16 00:55:22,623 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.Compactor(207): Compacting 2ed4b45335904963b9fb9d185fb956b0, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=104, earliestPutTs=1681606521569 2023-04-16 00:55:22,654 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] throttle.PressureAwareThroughputController(145): ee2f682ad6dd0fe8e91c021d54d3bdc5#B#compaction#331 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-16 00:55:22,662 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] throttle.PressureAwareThroughputController(145): ee2f682ad6dd0fe8e91c021d54d3bdc5#A#compaction#330 average throughput is 0.90 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-16 00:55:22,712 DEBUG [Thread-2414] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:55:22,712 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:55:22,714 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:55:22,716 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:55:22,716 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:55:22,716 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:55:22,716 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:22,716 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:55:22,716 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:55:22,717 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:22,717 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:22,718 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:22,719 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:22,719 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:55:22,720 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:22,720 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:22,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:22,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:22,721 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:22,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:55:22,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:22,721 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:55:22,721 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:55:22,721 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:55:22,721 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:55:22,721 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:22,721 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:55:22,722 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:55:22,722 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:55:22,722 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. 2023-04-16 00:55:22,722 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. started... 2023-04-16 00:55:22,722 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing ee2f682ad6dd0fe8e91c021d54d3bdc5 3/3 column families, dataSize=127.47 KB heapSize=365.06 KB 2023-04-16 00:55:22,722 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ee2f682ad6dd0fe8e91c021d54d3bdc5, store=A 2023-04-16 00:55:22,722 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 00:55:22,722 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ee2f682ad6dd0fe8e91c021d54d3bdc5, store=B 2023-04-16 00:55:22,722 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 00:55:22,722 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ee2f682ad6dd0fe8e91c021d54d3bdc5, store=C 2023-04-16 00:55:22,722 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 00:55:22,744 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304167b9587ae906a4ff5849a7b83aafc68ba to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304167b9587ae906a4ff5849a7b83aafc68ba 2023-04-16 00:55:22,757 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=140, memsize=42.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/.tmp/A/932b3720baa94283b6e6b897b5ac522f 2023-04-16 00:55:22,779 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=140 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/.tmp/B/054b59a0405246e8b8017056a77f2352 2023-04-16 00:55:22,798 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=140 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/.tmp/C/49b65699a7b04bf291d28f7195af0c98 2023-04-16 00:55:22,804 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/.tmp/A/932b3720baa94283b6e6b897b5ac522f as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/A/932b3720baa94283b6e6b897b5ac522f 2023-04-16 00:55:22,809 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/A/932b3720baa94283b6e6b897b5ac522f, entries=150, sequenceid=140, filesize=25.3 K 2023-04-16 00:55:22,810 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/.tmp/B/054b59a0405246e8b8017056a77f2352 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/B/054b59a0405246e8b8017056a77f2352 2023-04-16 00:55:22,813 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/B/054b59a0405246e8b8017056a77f2352, entries=150, sequenceid=140, filesize=11.8 K 2023-04-16 00:55:22,814 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/.tmp/C/49b65699a7b04bf291d28f7195af0c98 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/C/49b65699a7b04bf291d28f7195af0c98 2023-04-16 00:55:22,831 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/C/49b65699a7b04bf291d28f7195af0c98, entries=150, sequenceid=140, filesize=11.8 K 2023-04-16 00:55:22,832 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~365.02 KB/373776, currentSize=0 B/0 for ee2f682ad6dd0fe8e91c021d54d3bdc5 in 110ms, sequenceid=140, compaction requested=true 2023-04-16 00:55:22,832 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for ee2f682ad6dd0fe8e91c021d54d3bdc5: 2023-04-16 00:55:22,832 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. 2023-04-16 00:55:22,832 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:55:22,832 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:55:22,832 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:22,832 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:55:22,832 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:55:22,834 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:22,834 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:22,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:22,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:22,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:22,834 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:22,835 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:55:22,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:22,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:22,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:22,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:22,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:22,837 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:55:22,837 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@29a82067[Count = 0] remaining members to acquire global barrier 2023-04-16 00:55:22,837 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:55:22,837 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:22,839 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:22,839 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:22,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:22,840 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:55:22,840 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:55:22,840 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:55:22,840 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:22,840 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:55:22,841 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:22,841 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:55:22,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:22,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:22,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:22,841 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:22,841 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:55:22,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:22,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:22,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:22,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:22,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:22,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:22,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:22,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:55:22,845 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:55:22,845 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:55:22,846 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:55:22,846 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:55:22,846 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:55:22,846 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:55:22,847 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:22,847 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:22,847 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:22,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:22,847 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:22,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:22,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:22,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:22,848 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:22,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:22,848 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:22,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:22,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:22,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:22,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:22,859 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:22,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:22,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:22,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:22,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:22,866 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:22,866 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:22,866 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:22,866 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:22,866 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:22,866 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:22,866 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:22,866 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:22,866 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:22,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:22,866 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:22,866 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:55:22,867 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:55:22,866 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:22,867 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:22,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:22,867 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:55:22,867 DEBUG [Thread-2414] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:55:22,867 DEBUG [Thread-2414] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:55:22,967 DEBUG [Thread-2414] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:55:22,968 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:55:22,969 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:55:22,972 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:55:22,973 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:55:22,973 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:55:22,973 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:22,973 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:55:22,973 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:55:22,973 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:22,973 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:22,976 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:22,976 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:22,976 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:22,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:22,976 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:22,976 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:55:22,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:22,977 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:22,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:55:22,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:22,981 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:55:22,984 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:55:22,984 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:55:22,984 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:55:22,984 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:22,984 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:55:22,984 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:55:22,984 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:55:22,984 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. 2023-04-16 00:55:22,984 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. started... 2023-04-16 00:55:22,985 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing ee2f682ad6dd0fe8e91c021d54d3bdc5 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-04-16 00:55:22,985 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ee2f682ad6dd0fe8e91c021d54d3bdc5, store=A 2023-04-16 00:55:22,985 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:55:22,985 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ee2f682ad6dd0fe8e91c021d54d3bdc5, store=B 2023-04-16 00:55:22,985 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:55:22,985 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ee2f682ad6dd0fe8e91c021d54d3bdc5, store=C 2023-04-16 00:55:22,985 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:55:23,014 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(9158): Flush requested on ee2f682ad6dd0fe8e91c021d54d3bdc5 2023-04-16 00:55:23,014 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. as already flushing 2023-04-16 00:55:23,045 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681606523023, max timestamp=1681606523044Num uniques 859; 2023-04-16 00:55:23,046 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681606523023, max timestamp=1681606523044Num uniques 859; 2023-04-16 00:55:23,046 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681606523023, max timestamp=1681606523044Num uniques 859; 2023-04-16 00:55:23,046 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-16 00:55:23,046 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-16 00:55:23,046 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-16 00:55:23,068 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/.tmp/B/2b0f80526d1b4839b99cb342be941119 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/B/2b0f80526d1b4839b99cb342be941119 2023-04-16 00:55:23,073 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in ee2f682ad6dd0fe8e91c021d54d3bdc5/B of ee2f682ad6dd0fe8e91c021d54d3bdc5 into 2b0f80526d1b4839b99cb342be941119(size=11.8 K), total size for store is 23.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 00:55:23,073 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for ee2f682ad6dd0fe8e91c021d54d3bdc5: 2023-04-16 00:55:23,073 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5., storeName=ee2f682ad6dd0fe8e91c021d54d3bdc5/B, priority=11, startTime=1681606522617; duration=0sec 2023-04-16 00:55:23,073 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 00:55:23,074 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-16 00:55:23,076 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 71466 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-16 00:55:23,076 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HStore(1912): ee2f682ad6dd0fe8e91c021d54d3bdc5/C is initiating minor compaction (all files) 2023-04-16 00:55:23,076 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HRegion(2259): Starting compaction of ee2f682ad6dd0fe8e91c021d54d3bdc5/C in TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. 2023-04-16 00:55:23,077 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/C/71299a44e639499e8a3034bde1608367, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/C/fbdd963920664561b5cbe6b2544fc6c3, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/C/dee497a3a72947de84df13174cb2a796, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/C/9e1ac9a463654bab82cbfd048ba6294f, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/C/ada9cb30cf754d93858849098089397e, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/C/49b65699a7b04bf291d28f7195af0c98] into tmpdir=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/.tmp, totalSize=69.8 K 2023-04-16 00:55:23,077 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting 71299a44e639499e8a3034bde1608367, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=15, earliestPutTs=1681606516876 2023-04-16 00:55:23,077 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting fbdd963920664561b5cbe6b2544fc6c3, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=47, earliestPutTs=1681606516912 2023-04-16 00:55:23,078 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting dee497a3a72947de84df13174cb2a796, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=58, earliestPutTs=1681606519057 2023-04-16 00:55:23,078 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting 9e1ac9a463654bab82cbfd048ba6294f, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=93, earliestPutTs=1681606519417 2023-04-16 00:55:23,079 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting ada9cb30cf754d93858849098089397e, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=104, earliestPutTs=1681606521569 2023-04-16 00:55:23,079 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting 49b65699a7b04bf291d28f7195af0c98, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=140, earliestPutTs=1681606521759 2023-04-16 00:55:23,097 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] throttle.PressureAwareThroughputController(145): ee2f682ad6dd0fe8e91c021d54d3bdc5#C#compaction#336 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-16 00:55:23,101 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/.tmp/A/2071d651f31a4115bfe4039c27575f60 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/A/2071d651f31a4115bfe4039c27575f60 2023-04-16 00:55:23,108 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/.tmp/C/5cd6c75d70c8467c9df564e03dd709f2 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/C/5cd6c75d70c8467c9df564e03dd709f2 2023-04-16 00:55:23,111 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in ee2f682ad6dd0fe8e91c021d54d3bdc5/C of ee2f682ad6dd0fe8e91c021d54d3bdc5 into 5cd6c75d70c8467c9df564e03dd709f2(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 00:55:23,111 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in ee2f682ad6dd0fe8e91c021d54d3bdc5/A of ee2f682ad6dd0fe8e91c021d54d3bdc5 into 2071d651f31a4115bfe4039c27575f60(size=25.3 K), total size for store is 50.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 00:55:23,111 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for ee2f682ad6dd0fe8e91c021d54d3bdc5: 2023-04-16 00:55:23,111 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for ee2f682ad6dd0fe8e91c021d54d3bdc5: 2023-04-16 00:55:23,111 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5., storeName=ee2f682ad6dd0fe8e91c021d54d3bdc5/C, priority=10, startTime=1681606522618; duration=0sec 2023-04-16 00:55:23,111 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5., storeName=ee2f682ad6dd0fe8e91c021d54d3bdc5/A, priority=11, startTime=1681606522617; duration=0sec 2023-04-16 00:55:23,112 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 00:55:23,112 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 00:55:23,131 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=318, cellSize=14558, totalHeapSize=33312, min timestamp=1681606523074, max timestamp=1681606523128Num uniques 1068; 2023-04-16 00:55:23,131 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=318, cellSize=14558, totalHeapSize=33312, min timestamp=1681606523074, max timestamp=1681606523128Num uniques 1068; 2023-04-16 00:55:23,131 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=14558, suffix heap size=139040,new segment heap size=33312 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=318 2023-04-16 00:55:23,131 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=14558, suffix heap size=139040,new segment heap size=33312 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=318 2023-04-16 00:55:23,131 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=318, cellSize=14558, totalHeapSize=33312, min timestamp=1681606523074, max timestamp=1681606523128Num uniques 1068; 2023-04-16 00:55:23,131 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=14558, suffix heap size=139040,new segment heap size=33312 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=318 2023-04-16 00:55:23,173 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1681606523146, max timestamp=1681606523171Num uniques 1177; 2023-04-16 00:55:23,173 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=53902, new segment data size=10392, suffix heap size=150376,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1177, new segment cells count=227 2023-04-16 00:55:23,173 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1681606523146, max timestamp=1681606523171Num uniques 1177; 2023-04-16 00:55:23,174 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=53902, new segment data size=10392, suffix heap size=150376,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1177, new segment cells count=227 2023-04-16 00:55:23,174 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1681606523146, max timestamp=1681606523171Num uniques 1177; 2023-04-16 00:55:23,174 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=53902, new segment data size=10392, suffix heap size=150376,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1177, new segment cells count=227 2023-04-16 00:55:23,209 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=286, cellSize=13096, totalHeapSize=29984, min timestamp=1681606523197, max timestamp=1681606523207Num uniques 1086; 2023-04-16 00:55:23,210 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=13096, suffix heap size=140912,new segment heap size=29984 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=286 2023-04-16 00:55:23,210 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=286, cellSize=13096, totalHeapSize=29984, min timestamp=1681606523197, max timestamp=1681606523207Num uniques 1086; 2023-04-16 00:55:23,210 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=13096, suffix heap size=140912,new segment heap size=29984 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=286 2023-04-16 00:55:23,211 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=286, cellSize=13096, totalHeapSize=29984, min timestamp=1681606523197, max timestamp=1681606523207Num uniques 1086; 2023-04-16 00:55:23,211 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=13096, suffix heap size=140912,new segment heap size=29984 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=286 2023-04-16 00:55:23,249 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=345, cellSize=15800, totalHeapSize=36120, min timestamp=1681606523232, max timestamp=1681606523247Num uniques 1145; 2023-04-16 00:55:23,249 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=52440, new segment data size=15800, suffix heap size=147048,new segment heap size=36120 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1145, new segment cells count=345 2023-04-16 00:55:23,249 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=345, cellSize=15800, totalHeapSize=36120, min timestamp=1681606523232, max timestamp=1681606523247Num uniques 1145; 2023-04-16 00:55:23,249 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=52440, new segment data size=15800, suffix heap size=147048,new segment heap size=36120 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1145, new segment cells count=345 2023-04-16 00:55:23,250 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=345, cellSize=15800, totalHeapSize=36120, min timestamp=1681606523232, max timestamp=1681606523247Num uniques 1145; 2023-04-16 00:55:23,250 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=52440, new segment data size=15800, suffix heap size=147048,new segment heap size=36120 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1145, new segment cells count=345 2023-04-16 00:55:23,293 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:23,293 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59666 deadline: 1681606583284, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:23,293 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:23,293 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59642 deadline: 1681606583285, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:23,293 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:23,293 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59664 deadline: 1681606583286, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:23,294 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:23,294 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59650 deadline: 1681606583287, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:23,396 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:23,397 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59666 deadline: 1681606583394, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:23,397 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:23,397 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59642 deadline: 1681606583394, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:23,397 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:23,397 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59664 deadline: 1681606583394, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:23,402 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023041606bdb7a505f64291b3e8bf3e8642aa97 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041606bdb7a505f64291b3e8bf3e8642aa97 2023-04-16 00:55:23,405 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:23,405 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59650 deadline: 1681606583395, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:23,603 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:23,603 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59666 deadline: 1681606583598, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:23,604 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:23,604 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59642 deadline: 1681606583599, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:23,604 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:23,604 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59664 deadline: 1681606583599, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:23,604 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:23,604 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59592 deadline: 1681606583601, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:23,606 DEBUG [Thread-2406] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4177 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5., hostname=jenkins-hbase4.apache.org,34975,1681606419217, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor32.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-16 00:55:23,612 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:23,612 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59650 deadline: 1681606583607, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:23,807 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=147, memsize=8.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/.tmp/A/182cec2bc8564e7488bff17b333a205f 2023-04-16 00:55:23,819 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=147 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/.tmp/B/7d50407fb9fe4c1e902f8ded55f85a01 2023-04-16 00:55:23,909 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:23,909 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59664 deadline: 1681606583905, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:23,910 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:23,910 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59642 deadline: 1681606583905, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:23,912 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:23,912 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59666 deadline: 1681606583906, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:23,918 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:23,918 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59650 deadline: 1681606583915, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:24,232 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=147 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/.tmp/C/f78762c94569489a86c3607be4af2a47 2023-04-16 00:55:24,237 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/.tmp/A/182cec2bc8564e7488bff17b333a205f as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/A/182cec2bc8564e7488bff17b333a205f 2023-04-16 00:55:24,240 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/A/182cec2bc8564e7488bff17b333a205f, entries=200, sequenceid=147, filesize=32.1 K 2023-04-16 00:55:24,241 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/.tmp/B/7d50407fb9fe4c1e902f8ded55f85a01 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/B/7d50407fb9fe4c1e902f8ded55f85a01 2023-04-16 00:55:24,249 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/B/7d50407fb9fe4c1e902f8ded55f85a01, entries=150, sequenceid=147, filesize=11.8 K 2023-04-16 00:55:24,250 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/.tmp/C/f78762c94569489a86c3607be4af2a47 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/C/f78762c94569489a86c3607be4af2a47 2023-04-16 00:55:24,254 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/C/f78762c94569489a86c3607be4af2a47, entries=150, sequenceid=147, filesize=11.8 K 2023-04-16 00:55:24,255 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=161.02 KB/164880 for ee2f682ad6dd0fe8e91c021d54d3bdc5 in 1271ms, sequenceid=147, compaction requested=true 2023-04-16 00:55:24,255 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for ee2f682ad6dd0fe8e91c021d54d3bdc5: 2023-04-16 00:55:24,255 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. 2023-04-16 00:55:24,255 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:55:24,255 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:55:24,255 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:24,255 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:55:24,255 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:55:24,257 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:24,257 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:24,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:24,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:24,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:24,257 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:24,257 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:55:24,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:24,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:24,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:24,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:24,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:24,259 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:55:24,259 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@702cc5d5[Count = 0] remaining members to acquire global barrier 2023-04-16 00:55:24,259 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:55:24,259 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:24,260 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:24,260 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:24,260 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:24,260 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:55:24,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:24,260 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:55:24,260 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:55:24,260 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:55:24,262 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:55:24,262 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:24,262 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:55:24,262 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:24,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:24,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:24,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:24,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:24,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:24,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:24,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:24,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:24,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:24,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:24,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:55:24,267 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:55:24,267 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:55:24,267 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:55:24,267 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:55:24,267 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:55:24,267 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:55:24,270 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:24,270 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:24,270 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:24,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:24,270 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:24,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:24,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:24,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:24,271 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:24,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:24,271 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:24,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:24,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:24,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:24,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:24,272 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:24,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:24,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:24,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:24,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:24,274 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:24,274 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:24,274 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:24,274 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:24,274 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:24,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:24,275 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:24,274 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:24,275 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:55:24,275 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:55:24,275 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:24,275 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:24,275 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:24,275 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:24,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:24,275 DEBUG [Thread-2414] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:55:24,275 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:24,275 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:55:24,275 DEBUG [Thread-2414] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:55:24,375 DEBUG [Thread-2414] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:55:24,376 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:55:24,377 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:55:24,379 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:55:24,379 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:55:24,379 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:55:24,379 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:24,379 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:55:24,380 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:55:24,380 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:24,380 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:24,381 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:24,381 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:24,381 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:24,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:24,382 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:24,382 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:55:24,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:24,382 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:24,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:55:24,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:24,382 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:55:24,383 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:55:24,383 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:55:24,383 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:55:24,383 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:24,383 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:55:24,383 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:55:24,383 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:55:24,383 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. 2023-04-16 00:55:24,383 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. started... 2023-04-16 00:55:24,383 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing ee2f682ad6dd0fe8e91c021d54d3bdc5 3/3 column families, dataSize=161.02 KB heapSize=447.23 KB 2023-04-16 00:55:24,384 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ee2f682ad6dd0fe8e91c021d54d3bdc5, store=A 2023-04-16 00:55:24,384 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 00:55:24,384 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ee2f682ad6dd0fe8e91c021d54d3bdc5, store=B 2023-04-16 00:55:24,384 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 00:55:24,384 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ee2f682ad6dd0fe8e91c021d54d3bdc5, store=C 2023-04-16 00:55:24,384 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 00:55:24,417 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230416df80ceb84edf48f48ec6fbce63c178d5 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230416df80ceb84edf48f48ec6fbce63c178d5 2023-04-16 00:55:24,419 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(9158): Flush requested on ee2f682ad6dd0fe8e91c021d54d3bdc5 2023-04-16 00:55:24,419 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. as already flushing 2023-04-16 00:55:24,427 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=256, memsize=53.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/.tmp/A/678fac82f9f74535baae96803b8563de 2023-04-16 00:55:24,433 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:24,433 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59642 deadline: 1681606584422, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:24,433 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:24,433 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59664 deadline: 1681606584423, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:24,433 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:24,433 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59666 deadline: 1681606584423, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:24,439 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=53.67 KB at sequenceid=256 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/.tmp/B/2c02ceba33224305bed1757852beaca3 2023-04-16 00:55:24,439 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:24,439 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59650 deadline: 1681606584432, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:24,539 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:24,539 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59642 deadline: 1681606584534, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:24,540 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:24,540 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59664 deadline: 1681606584534, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:24,540 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:24,540 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59666 deadline: 1681606584534, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:24,545 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:24,545 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59650 deadline: 1681606584540, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:24,747 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:24,747 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59664 deadline: 1681606584741, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:24,747 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:24,747 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59642 deadline: 1681606584741, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:24,747 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:24,747 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59666 deadline: 1681606584742, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:24,751 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:24,751 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59650 deadline: 1681606584746, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:24,853 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=53.67 KB at sequenceid=256 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/.tmp/C/29c995d0bad14ddeb5ac50be29be5566 2023-04-16 00:55:24,857 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/.tmp/A/678fac82f9f74535baae96803b8563de as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/A/678fac82f9f74535baae96803b8563de 2023-04-16 00:55:24,861 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/A/678fac82f9f74535baae96803b8563de, entries=150, sequenceid=256, filesize=25.3 K 2023-04-16 00:55:24,862 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/.tmp/B/2c02ceba33224305bed1757852beaca3 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/B/2c02ceba33224305bed1757852beaca3 2023-04-16 00:55:24,886 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/B/2c02ceba33224305bed1757852beaca3, entries=150, sequenceid=256, filesize=11.8 K 2023-04-16 00:55:24,887 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/.tmp/C/29c995d0bad14ddeb5ac50be29be5566 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/C/29c995d0bad14ddeb5ac50be29be5566 2023-04-16 00:55:24,906 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/C/29c995d0bad14ddeb5ac50be29be5566, entries=150, sequenceid=256, filesize=11.8 K 2023-04-16 00:55:24,906 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~161.02 KB/164880, heapSize ~447.19 KB/457920, currentSize=26.84 KB/27480 for ee2f682ad6dd0fe8e91c021d54d3bdc5 in 523ms, sequenceid=256, compaction requested=true 2023-04-16 00:55:24,907 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for ee2f682ad6dd0fe8e91c021d54d3bdc5: 2023-04-16 00:55:24,907 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. 2023-04-16 00:55:24,907 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:55:24,907 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:55:24,907 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:24,908 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:55:24,908 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:55:24,913 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:24,913 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:24,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:24,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:24,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:24,913 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:24,913 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:55:24,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:24,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:24,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:24,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:24,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:24,915 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:55:24,915 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@dbecbc7[Count = 0] remaining members to acquire global barrier 2023-04-16 00:55:24,915 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:55:24,915 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:24,922 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:24,922 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:24,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:24,922 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:55:24,922 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:55:24,922 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:55:24,923 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:24,923 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:55:24,923 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:24,924 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:55:24,923 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:24,923 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:55:24,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:24,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:24,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:24,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:24,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:24,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:24,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:24,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:24,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:24,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:24,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:55:24,926 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:55:24,926 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:55:24,927 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:55:24,927 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:55:24,927 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:55:24,927 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:55:24,928 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:24,928 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:24,928 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:24,928 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:24,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:24,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:24,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:24,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:24,929 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:24,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:24,929 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:24,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:24,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:24,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:24,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:24,930 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:24,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:24,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:24,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:24,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:24,940 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:24,940 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:24,940 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:24,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:24,940 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:24,940 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:24,940 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:24,940 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:55:24,940 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:55:24,940 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:24,940 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:24,940 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:24,940 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:24,940 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:24,940 DEBUG [Thread-2414] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:55:24,940 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:24,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:24,940 DEBUG [Thread-2414] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:55:24,940 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:55:25,041 DEBUG [Thread-2414] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:55:25,041 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:55:25,043 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:55:25,048 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:55:25,048 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:55:25,048 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:55:25,048 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:25,048 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:55:25,049 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:55:25,049 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:25,049 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:25,051 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:25,051 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:25,052 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:25,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:25,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:25,053 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:25,053 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:55:25,053 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:25,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:55:25,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:25,054 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:55:25,054 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:55:25,054 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:55:25,054 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:55:25,054 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:25,054 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:55:25,054 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:55:25,054 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:55:25,054 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. 2023-04-16 00:55:25,054 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. started... 2023-04-16 00:55:25,055 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing ee2f682ad6dd0fe8e91c021d54d3bdc5 3/3 column families, dataSize=33.54 KB heapSize=100.36 KB 2023-04-16 00:55:25,055 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ee2f682ad6dd0fe8e91c021d54d3bdc5, store=A 2023-04-16 00:55:25,055 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:55:25,055 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ee2f682ad6dd0fe8e91c021d54d3bdc5, store=B 2023-04-16 00:55:25,055 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:55:25,055 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ee2f682ad6dd0fe8e91c021d54d3bdc5, store=C 2023-04-16 00:55:25,055 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:55:25,056 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(9158): Flush requested on ee2f682ad6dd0fe8e91c021d54d3bdc5 2023-04-16 00:55:25,057 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. as already flushing 2023-04-16 00:55:25,095 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1681606525074, max timestamp=1681606525092Num uniques 859; 2023-04-16 00:55:25,095 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1681606525074, max timestamp=1681606525092Num uniques 859; 2023-04-16 00:55:25,095 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1681606525074, max timestamp=1681606525092Num uniques 859; 2023-04-16 00:55:25,095 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-16 00:55:25,095 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-16 00:55:25,095 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-16 00:55:25,096 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230416bc17d0e51c5c45c2a2b357d4447c3cfd to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230416bc17d0e51c5c45c2a2b357d4447c3cfd 2023-04-16 00:55:25,144 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:25,144 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59666 deadline: 1681606585136, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:25,146 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1681606525123, max timestamp=1681606525142Num uniques 1168; 2023-04-16 00:55:25,146 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=53488, new segment data size=12268, suffix heap size=149440,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1168, new segment cells count=268 2023-04-16 00:55:25,146 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1681606525123, max timestamp=1681606525142Num uniques 1168; 2023-04-16 00:55:25,146 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=53488, new segment data size=12268, suffix heap size=149440,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1168, new segment cells count=268 2023-04-16 00:55:25,146 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1681606525123, max timestamp=1681606525142Num uniques 1168; 2023-04-16 00:55:25,146 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=53488, new segment data size=12268, suffix heap size=149440,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1168, new segment cells count=268 2023-04-16 00:55:25,195 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:25,195 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59664 deadline: 1681606585190, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:25,203 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:25,203 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59642 deadline: 1681606585195, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:25,203 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:25,203 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59650 deadline: 1681606585195, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:25,263 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:25,263 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59666 deadline: 1681606585253, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:25,307 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:25,307 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59664 deadline: 1681606585296, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:25,308 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:25,308 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59642 deadline: 1681606585304, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:25,308 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:25,308 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59650 deadline: 1681606585305, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:25,470 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:25,470 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59666 deadline: 1681606585466, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:25,511 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=264, memsize=11.2 K, hasBloomFilter=true, into tmp file hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/.tmp/A/fcb8dc602cda460989697a888a026465 2023-04-16 00:55:25,516 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:25,516 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59664 deadline: 1681606585509, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:25,516 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:25,516 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59642 deadline: 1681606585510, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:25,516 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:25,516 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59650 deadline: 1681606585510, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:25,782 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:25,783 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59666 deadline: 1681606585772, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:25,826 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:25,826 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59664 deadline: 1681606585817, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:25,826 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:25,826 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59650 deadline: 1681606585817, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:25,826 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:25,826 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59642 deadline: 1681606585818, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:25,927 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=264 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/.tmp/B/67320ea88b954a078c0608b9799b3e08 2023-04-16 00:55:26,292 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:26,292 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59666 deadline: 1681606586287, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:26,331 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:26,331 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59664 deadline: 1681606586327, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:26,333 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:26,333 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59650 deadline: 1681606586329, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:26,334 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:26,334 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59642 deadline: 1681606586329, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:26,337 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=264 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/.tmp/C/0065cc7bde58460e98227dcbec7c11cc 2023-04-16 00:55:26,341 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/.tmp/A/fcb8dc602cda460989697a888a026465 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/A/fcb8dc602cda460989697a888a026465 2023-04-16 00:55:26,344 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/A/fcb8dc602cda460989697a888a026465, entries=150, sequenceid=264, filesize=25.4 K 2023-04-16 00:55:26,345 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/.tmp/B/67320ea88b954a078c0608b9799b3e08 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/B/67320ea88b954a078c0608b9799b3e08 2023-04-16 00:55:26,348 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/B/67320ea88b954a078c0608b9799b3e08, entries=150, sequenceid=264, filesize=11.9 K 2023-04-16 00:55:26,349 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/.tmp/C/0065cc7bde58460e98227dcbec7c11cc as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/C/0065cc7bde58460e98227dcbec7c11cc 2023-04-16 00:55:26,352 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/C/0065cc7bde58460e98227dcbec7c11cc, entries=150, sequenceid=264, filesize=11.9 K 2023-04-16 00:55:26,353 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~100.31 KB/102720, currentSize=147.60 KB/151140 for ee2f682ad6dd0fe8e91c021d54d3bdc5 in 1298ms, sequenceid=264, compaction requested=true 2023-04-16 00:55:26,353 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for ee2f682ad6dd0fe8e91c021d54d3bdc5: 2023-04-16 00:55:26,353 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. 2023-04-16 00:55:26,353 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:55:26,353 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:55:26,353 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:26,353 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:55:26,353 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:55:26,354 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:26,354 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:26,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:26,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:26,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:26,355 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:26,355 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:55:26,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:26,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:26,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:26,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:26,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:26,356 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:55:26,356 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@ca6da3b[Count = 0] remaining members to acquire global barrier 2023-04-16 00:55:26,356 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:55:26,356 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:26,357 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:26,357 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:26,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:26,357 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:55:26,357 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:55:26,357 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:55:26,357 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:26,357 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:55:26,359 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:55:26,359 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:26,359 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:26,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:26,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:26,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:26,359 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:55:26,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:26,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:26,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:26,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:26,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:26,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:26,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:26,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:55:26,361 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:55:26,361 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:55:26,361 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:55:26,361 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:55:26,361 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:55:26,361 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:55:26,363 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:26,363 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:26,363 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:26,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:26,363 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:26,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:26,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:26,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:26,364 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:26,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:26,364 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:26,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:26,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:26,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:26,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:26,365 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:26,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:26,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:26,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:26,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:26,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:26,367 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:26,367 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:26,367 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:26,367 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:26,367 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:55:26,367 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:55:26,367 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:26,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:26,367 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:26,367 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:26,367 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:26,367 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:55:26,367 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:26,367 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:26,367 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:26,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:26,367 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:26,367 DEBUG [Thread-2414] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:55:26,368 DEBUG [Thread-2414] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:55:26,468 DEBUG [Thread-2414] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:55:26,468 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:55:26,470 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:55:26,471 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:55:26,471 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:55:26,471 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:55:26,471 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:26,471 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:55:26,471 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:55:26,472 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:26,472 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:26,473 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:26,473 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:26,473 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:26,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:26,473 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:26,473 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:55:26,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:26,473 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:26,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:55:26,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:26,474 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:55:26,474 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:55:26,474 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:55:26,474 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:55:26,474 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:26,474 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:55:26,474 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:55:26,474 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:55:26,474 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. 2023-04-16 00:55:26,474 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. started... 2023-04-16 00:55:26,475 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing ee2f682ad6dd0fe8e91c021d54d3bdc5 3/3 column families, dataSize=147.60 KB heapSize=414.61 KB 2023-04-16 00:55:26,475 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ee2f682ad6dd0fe8e91c021d54d3bdc5, store=A 2023-04-16 00:55:26,475 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 00:55:26,475 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ee2f682ad6dd0fe8e91c021d54d3bdc5, store=B 2023-04-16 00:55:26,475 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 00:55:26,475 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ee2f682ad6dd0fe8e91c021d54d3bdc5, store=C 2023-04-16 00:55:26,475 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 00:55:26,486 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230416223d6d34841f4bb7a437f41ee93e8f70 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230416223d6d34841f4bb7a437f41ee93e8f70 2023-04-16 00:55:26,891 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=318, memsize=49.2 K, hasBloomFilter=true, into tmp file hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/.tmp/A/8ac9840dd9f84421b70650c911dd55c3 2023-04-16 00:55:26,900 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=318 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/.tmp/B/578d5bb5951a4745b077be16e0997cfa 2023-04-16 00:55:26,910 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=318 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/.tmp/C/89572d68c50148d89d56409b4eab3f9b 2023-04-16 00:55:26,913 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/.tmp/A/8ac9840dd9f84421b70650c911dd55c3 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/A/8ac9840dd9f84421b70650c911dd55c3 2023-04-16 00:55:26,919 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/A/8ac9840dd9f84421b70650c911dd55c3, entries=150, sequenceid=318, filesize=25.4 K 2023-04-16 00:55:26,920 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/.tmp/B/578d5bb5951a4745b077be16e0997cfa as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/B/578d5bb5951a4745b077be16e0997cfa 2023-04-16 00:55:26,923 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/B/578d5bb5951a4745b077be16e0997cfa, entries=150, sequenceid=318, filesize=11.9 K 2023-04-16 00:55:26,923 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/.tmp/C/89572d68c50148d89d56409b4eab3f9b as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/C/89572d68c50148d89d56409b4eab3f9b 2023-04-16 00:55:26,929 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/C/89572d68c50148d89d56409b4eab3f9b, entries=150, sequenceid=318, filesize=11.9 K 2023-04-16 00:55:26,930 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~147.60 KB/151140, heapSize ~414.56 KB/424512, currentSize=0 B/0 for ee2f682ad6dd0fe8e91c021d54d3bdc5 in 455ms, sequenceid=318, compaction requested=true 2023-04-16 00:55:26,930 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for ee2f682ad6dd0fe8e91c021d54d3bdc5: 2023-04-16 00:55:26,930 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. 2023-04-16 00:55:26,930 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:55:26,930 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:55:26,930 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:26,930 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:55:26,930 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:55:26,932 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:26,932 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:26,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:26,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:26,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:26,932 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:26,932 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:55:26,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:26,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:26,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:26,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:26,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:26,934 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:55:26,934 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@64e568d0[Count = 0] remaining members to acquire global barrier 2023-04-16 00:55:26,934 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:55:26,934 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:26,940 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:26,940 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:26,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:26,940 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:55:26,940 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:55:26,940 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:26,940 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:55:26,940 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:55:26,941 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:26,941 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:55:26,941 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:26,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:26,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:26,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:26,941 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:55:26,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:26,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:26,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:26,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:26,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:26,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:26,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:26,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:55:26,948 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:55:26,948 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:55:26,948 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:55:26,948 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:55:26,948 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:55:26,948 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:55:26,957 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:26,957 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:26,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:26,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:26,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:26,957 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:26,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:26,957 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:26,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:26,957 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:26,957 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:26,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:26,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:26,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:26,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:26,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:26,959 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:26,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:26,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:26,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:26,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:26,961 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:26,961 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:26,961 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:26,961 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:26,961 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:26,961 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:26,961 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:55:26,961 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:55:26,961 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:26,961 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:26,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:26,961 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:26,961 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:26,961 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:26,961 DEBUG [Thread-2414] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:55:26,961 DEBUG [Thread-2414] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:55:26,962 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:55:26,962 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:26,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:27,062 DEBUG [Thread-2414] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:55:27,062 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:55:27,064 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:55:27,065 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:55:27,065 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:55:27,065 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:55:27,065 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:27,065 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:55:27,065 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:55:27,066 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:27,066 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:27,068 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:27,068 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:27,068 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:27,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:27,069 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:27,069 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:55:27,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:27,069 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:27,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:55:27,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:27,069 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:55:27,070 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:55:27,070 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:55:27,070 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:55:27,070 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:27,070 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:55:27,070 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:55:27,070 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:55:27,070 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. 2023-04-16 00:55:27,070 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. started... 2023-04-16 00:55:27,070 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for ee2f682ad6dd0fe8e91c021d54d3bdc5: 2023-04-16 00:55:27,070 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. 2023-04-16 00:55:27,070 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:55:27,070 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:55:27,070 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:27,070 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:55:27,070 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:55:27,072 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:27,072 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:27,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:27,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:27,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:27,072 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:27,072 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:55:27,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:27,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:27,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:27,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:27,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:27,074 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:55:27,075 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:55:27,075 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@72711bf4[Count = 0] remaining members to acquire global barrier 2023-04-16 00:55:27,075 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:27,076 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:27,076 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:27,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:27,076 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:55:27,076 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:55:27,076 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:55:27,081 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:55:27,081 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:27,081 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:27,081 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:55:27,081 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:55:27,081 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:55:27,081 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:55:27,082 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:55:27,082 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:55:27,082 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:55:27,082 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:55:27,083 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:27,083 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:27,083 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:27,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:27,083 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:27,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:27,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:27,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:27,084 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:27,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:27,084 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:27,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:27,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:27,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:27,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:27,085 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:27,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:27,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:27,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:27,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:27,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:27,087 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:27,087 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:27,087 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:27,087 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:27,087 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:27,087 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:55:27,087 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:27,087 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:55:27,087 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:27,088 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:55:27,087 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:27,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:27,088 DEBUG [Thread-2414] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:55:27,088 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:27,088 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:27,088 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:27,088 DEBUG [Thread-2414] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:55:27,088 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:27,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:27,188 DEBUG [Thread-2414] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:55:27,189 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:55:27,191 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:55:27,192 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:55:27,193 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:55:27,193 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:55:27,193 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:27,193 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:55:27,193 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:55:27,193 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:27,193 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:27,194 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:27,194 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:27,194 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:27,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:27,195 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:27,195 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:55:27,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:27,195 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:27,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:55:27,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:27,195 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:55:27,196 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:55:27,196 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:55:27,196 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:55:27,196 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:27,196 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:55:27,196 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:55:27,196 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:55:27,196 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. 2023-04-16 00:55:27,196 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. started... 2023-04-16 00:55:27,196 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for ee2f682ad6dd0fe8e91c021d54d3bdc5: 2023-04-16 00:55:27,196 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. 2023-04-16 00:55:27,197 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:55:27,197 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:55:27,197 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:27,197 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:55:27,197 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:55:27,199 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:27,199 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:27,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:27,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:27,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:27,199 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:27,199 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:55:27,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:27,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:27,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:27,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:27,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:27,201 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:55:27,201 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@96a068e[Count = 0] remaining members to acquire global barrier 2023-04-16 00:55:27,201 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:55:27,201 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:27,202 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:27,202 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:27,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:27,202 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:55:27,202 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:27,202 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:55:27,202 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:55:27,203 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:55:27,204 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:55:27,204 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:27,204 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:27,204 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:55:27,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:27,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:27,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:27,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:27,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:27,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:27,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:27,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:27,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:27,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:27,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:55:27,206 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:55:27,206 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:55:27,207 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:55:27,207 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:55:27,207 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:55:27,207 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:55:27,208 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:27,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:27,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:27,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:27,208 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:27,208 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:27,208 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:27,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:27,209 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:27,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:27,209 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:27,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:27,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:27,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:27,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:27,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:27,210 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:27,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:27,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:27,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:27,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:27,213 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:27,213 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:27,213 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:27,213 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:27,213 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:27,213 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:27,213 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:27,213 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:27,213 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:27,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:27,213 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:27,213 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:27,213 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:55:27,213 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:55:27,213 DEBUG [Thread-2414] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:55:27,213 DEBUG [Thread-2414] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:55:27,214 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:55:27,214 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:27,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:27,314 DEBUG [Thread-2414] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:55:27,314 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:55:27,316 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:55:27,317 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:55:27,317 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:55:27,317 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:55:27,317 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:27,317 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:55:27,317 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:55:27,318 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:27,318 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:27,319 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:27,319 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:27,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:27,319 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:27,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:27,320 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:27,320 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:27,320 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:55:27,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:55:27,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:27,321 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:55:27,322 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:55:27,322 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:55:27,322 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:55:27,322 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:27,322 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:55:27,323 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:55:27,323 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:55:27,323 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. 2023-04-16 00:55:27,323 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. started... 2023-04-16 00:55:27,323 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for ee2f682ad6dd0fe8e91c021d54d3bdc5: 2023-04-16 00:55:27,323 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. 2023-04-16 00:55:27,323 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:55:27,323 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:55:27,323 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:27,323 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:55:27,323 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:55:27,325 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:27,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:27,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:27,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:27,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:27,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:27,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:27,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:27,326 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:27,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:27,327 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:27,327 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:55:27,327 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:55:27,327 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:55:27,327 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3327629a[Count = 0] remaining members to acquire global barrier 2023-04-16 00:55:27,327 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:27,328 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:27,328 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:27,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:27,328 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:55:27,328 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:55:27,328 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:27,328 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:55:27,328 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:55:27,331 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:27,331 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:55:27,331 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:27,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:27,331 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:55:27,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:27,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:27,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:27,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:27,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:27,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:27,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:27,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:27,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:27,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:55:27,334 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:55:27,334 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:55:27,334 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:55:27,334 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:55:27,334 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:55:27,334 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:55:27,342 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:27,342 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:27,342 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:27,342 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:27,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:27,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:27,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:27,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:27,342 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:27,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:27,342 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:27,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:27,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:27,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:27,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:27,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:27,344 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:27,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:27,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:27,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:27,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:27,346 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:27,346 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:27,346 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:27,347 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:27,346 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:27,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:27,347 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:55:27,347 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:55:27,347 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:27,347 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:27,347 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:27,347 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:27,347 DEBUG [Thread-2414] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:55:27,347 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:55:27,347 DEBUG [Thread-2414] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:55:27,347 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:27,347 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:27,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:27,347 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:27,423 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing ee2f682ad6dd0fe8e91c021d54d3bdc5 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-16 00:55:27,423 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ee2f682ad6dd0fe8e91c021d54d3bdc5, store=A 2023-04-16 00:55:27,423 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:55:27,423 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ee2f682ad6dd0fe8e91c021d54d3bdc5, store=B 2023-04-16 00:55:27,423 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:55:27,423 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ee2f682ad6dd0fe8e91c021d54d3bdc5, store=C 2023-04-16 00:55:27,423 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:55:27,424 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(9158): Flush requested on ee2f682ad6dd0fe8e91c021d54d3bdc5 2023-04-16 00:55:27,447 DEBUG [Thread-2414] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:55:27,448 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:55:27,452 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:55:27,456 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:55:27,456 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:55:27,456 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:55:27,456 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:27,456 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:55:27,456 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:55:27,457 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:27,457 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:27,458 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:27,458 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:27,458 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:27,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:27,458 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:27,458 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:55:27,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:27,459 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:27,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:55:27,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:27,459 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:55:27,480 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:55:27,480 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:55:27,480 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:55:27,480 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:27,480 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:55:27,488 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:55:27,488 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:55:27,488 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. 2023-04-16 00:55:27,488 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. started... 2023-04-16 00:55:27,488 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. as already flushing 2023-04-16 00:55:27,488 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. 2023-04-16 00:55:27,489 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:55:27,489 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:55:27,489 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:27,489 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:55:27,489 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:55:27,492 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:27,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:27,492 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:27,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:27,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:27,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:27,492 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:27,492 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:55:27,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:27,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:27,493 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230416f102adec4ffa43448e7bf5f94191795e to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230416f102adec4ffa43448e7bf5f94191795e 2023-04-16 00:55:27,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:27,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:27,493 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:55:27,493 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7c4b3fc6[Count = 0] remaining members to acquire global barrier 2023-04-16 00:55:27,493 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:55:27,494 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:27,495 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:27,495 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:55:27,495 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:27,495 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:27,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:27,495 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:55:27,495 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:55:27,495 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:55:27,497 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:27,497 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:55:27,497 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:27,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:27,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:27,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:27,497 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:55:27,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:27,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:27,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:27,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:27,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:27,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:27,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:27,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:55:27,499 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:55:27,499 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:55:27,499 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:55:27,499 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:55:27,499 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:55:27,499 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:55:27,504 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11854, totalHeapSize=27176, min timestamp=1681606527487, max timestamp=1681606527502Num uniques 859; 2023-04-16 00:55:27,504 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=11854, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-16 00:55:27,505 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11854, totalHeapSize=27176, min timestamp=1681606527487, max timestamp=1681606527502Num uniques 859; 2023-04-16 00:55:27,505 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=11854, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-16 00:55:27,505 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:27,505 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11854, totalHeapSize=27176, min timestamp=1681606527487, max timestamp=1681606527502Num uniques 859; 2023-04-16 00:55:27,505 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:27,505 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:27,505 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=11854, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-16 00:55:27,505 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:27,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:27,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:27,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:27,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:27,506 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:27,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:27,506 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:27,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:27,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:27,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:27,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:27,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:27,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:27,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:27,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:27,508 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:27,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:27,510 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:27,510 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:27,510 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:27,510 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:27,510 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:27,510 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:27,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:27,510 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:27,510 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:55:27,510 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:55:27,510 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:27,510 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:27,510 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:27,510 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:27,510 DEBUG [Thread-2414] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:55:27,510 DEBUG [Thread-2414] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:55:27,510 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:27,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:27,582 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:27,582 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59650 deadline: 1681606587557, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:27,583 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:27,583 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59666 deadline: 1681606587558, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:27,583 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:27,583 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59642 deadline: 1681606587558, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:27,585 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:27,585 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59664 deadline: 1681606587573, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:27,610 DEBUG [Thread-2414] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:55:27,611 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:55:27,615 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:55:27,624 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:55:27,624 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:55:27,624 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:55:27,624 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:27,624 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:55:27,624 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:55:27,625 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:27,625 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:27,627 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:27,627 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:27,627 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:27,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:27,627 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:27,627 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:55:27,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:27,628 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:27,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:55:27,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:27,628 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:55:27,632 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:55:27,632 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:55:27,632 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:55:27,632 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:27,632 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:55:27,636 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:55:27,659 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. 2023-04-16 00:55:27,659 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:55:27,659 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. started... 2023-04-16 00:55:27,659 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. as already flushing 2023-04-16 00:55:27,659 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. 2023-04-16 00:55:27,659 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:55:27,659 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:55:27,659 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:27,659 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:55:27,659 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:55:27,660 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:27,660 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59592 deadline: 1681606587631, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:27,663 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:27,663 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:27,663 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:27,663 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:55:27,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:27,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:27,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:27,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:27,665 DEBUG [Thread-2406] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8234 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5., hostname=jenkins-hbase4.apache.org,34975,1681606419217, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor32.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-16 00:55:27,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:27,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:27,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:27,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:27,668 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:55:27,668 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@61f2500[Count = 0] remaining members to acquire global barrier 2023-04-16 00:55:27,668 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:55:27,668 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:27,670 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:27,670 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:27,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:27,670 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:55:27,670 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:55:27,670 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:55:27,670 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:27,670 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:55:27,673 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:55:27,673 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:27,673 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:55:27,673 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:27,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:27,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:27,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:27,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:27,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:27,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:27,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:27,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:27,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:27,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:27,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:55:27,680 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:55:27,680 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:55:27,680 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:55:27,680 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:55:27,680 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:55:27,680 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:55:27,682 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:27,682 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:27,682 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:27,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:27,683 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:27,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:27,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:27,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:27,683 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:27,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:27,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:27,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:27,684 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:27,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:27,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:27,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:27,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:27,687 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:27,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:27,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:27,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:27,690 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:27,690 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:27,690 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:27,690 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:27,690 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:55:27,690 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:55:27,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:27,690 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:27,690 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:27,690 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:27,690 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:27,690 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:27,690 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:27,690 DEBUG [Thread-2414] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:55:27,690 DEBUG [Thread-2414] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:55:27,690 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:27,690 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:27,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:27,690 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:55:27,694 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:27,694 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59650 deadline: 1681606587684, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:27,694 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:27,695 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59642 deadline: 1681606587684, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:27,695 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:27,695 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59666 deadline: 1681606587684, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:27,700 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:27,700 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59664 deadline: 1681606587686, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:27,790 DEBUG [Thread-2414] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:55:27,791 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:55:27,793 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:55:27,794 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:55:27,794 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:55:27,794 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:55:27,794 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:27,794 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:55:27,794 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:55:27,794 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:27,794 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:27,798 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:27,798 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:27,798 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:27,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:27,799 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:27,799 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:55:27,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:27,799 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:27,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:55:27,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:27,799 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:55:27,800 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:55:27,800 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:55:27,800 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:55:27,800 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:27,800 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:55:27,804 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:55:27,804 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:55:27,804 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. 2023-04-16 00:55:27,804 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. started... 2023-04-16 00:55:27,804 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. as already flushing 2023-04-16 00:55:27,804 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. 2023-04-16 00:55:27,804 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:55:27,804 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:55:27,804 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:27,804 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:55:27,804 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:55:27,805 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:27,806 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:27,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:27,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:27,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:27,806 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:27,806 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:55:27,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:27,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:27,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:27,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:27,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:27,807 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:55:27,807 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5f981d3b[Count = 0] remaining members to acquire global barrier 2023-04-16 00:55:27,807 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:55:27,808 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:27,809 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:27,809 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:27,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:27,809 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:55:27,809 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:55:27,809 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:55:27,809 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:27,809 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:55:27,810 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:27,810 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:55:27,810 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:27,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:27,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:27,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:27,810 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:55:27,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:27,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:27,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:27,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:27,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:27,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:27,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:27,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:55:27,812 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:55:27,812 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:55:27,812 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:55:27,812 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:55:27,812 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:55:27,812 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:55:27,815 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:27,815 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:27,815 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:27,815 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:27,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:27,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:27,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:27,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:27,815 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:27,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:27,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:27,815 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:27,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:27,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:27,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:27,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:27,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:27,816 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:27,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:27,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:27,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:27,818 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:27,818 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:27,818 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:27,818 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:27,819 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:55:27,819 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:55:27,818 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:27,819 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:55:27,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:27,819 DEBUG [Thread-2414] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:55:27,819 DEBUG [Thread-2414] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:55:27,818 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:27,818 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:27,819 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:27,819 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:27,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:27,819 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:27,819 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:27,819 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:27,903 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=331, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/.tmp/A/a1de6d30d39b472f8f371b2fe15e3c2a 2023-04-16 00:55:27,904 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:27,904 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59650 deadline: 1681606587895, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:27,904 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:27,905 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59642 deadline: 1681606587896, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:27,905 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:27,905 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59666 deadline: 1681606587897, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:27,909 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:27,910 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59664 deadline: 1681606587901, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:27,919 DEBUG [Thread-2414] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:55:27,919 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:55:27,921 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:55:27,923 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:55:27,923 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:55:27,926 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:55:27,926 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:27,926 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:55:27,926 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:55:27,926 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:27,926 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:27,927 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:27,928 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:27,928 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:27,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:27,928 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:27,928 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:55:27,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:27,928 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:27,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:55:27,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:27,929 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:55:27,929 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:55:27,929 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:55:27,929 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:55:27,929 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:27,929 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:55:27,929 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:55:27,929 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:55:27,929 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. 2023-04-16 00:55:27,929 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. started... 2023-04-16 00:55:27,930 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. as already flushing 2023-04-16 00:55:27,930 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. 2023-04-16 00:55:27,930 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:55:27,930 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:55:27,930 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:27,930 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:55:27,930 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:55:27,932 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:27,932 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:27,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:27,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:27,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:27,933 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:27,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:27,933 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:55:27,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:27,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:27,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:27,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:27,934 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:55:27,934 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:55:27,934 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@572397e0[Count = 0] remaining members to acquire global barrier 2023-04-16 00:55:27,934 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:27,936 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:27,936 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:27,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:27,936 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:55:27,936 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:55:27,936 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:55:27,936 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:27,936 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:55:27,937 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:27,937 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:55:27,937 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:27,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:27,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:27,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:27,937 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:55:27,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:27,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:27,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:27,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:27,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:27,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:27,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:27,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:55:27,941 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:55:27,941 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:55:27,941 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:55:27,941 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:55:27,941 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:55:27,942 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:55:27,943 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:27,943 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:27,943 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:27,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:27,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:27,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:27,943 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:27,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:27,944 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:27,944 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:27,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:27,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:27,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:27,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:27,946 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:27,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:27,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:27,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:27,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:27,952 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:27,952 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:27,952 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:27,952 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:27,952 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:55:27,952 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:55:27,952 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:27,952 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:27,952 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:27,952 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:27,952 DEBUG [Thread-2414] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:55:27,953 DEBUG [Thread-2414] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:55:27,953 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:27,953 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:27,953 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:27,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:27,954 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:55:27,954 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:27,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:28,053 DEBUG [Thread-2414] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:55:28,053 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:55:28,055 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:55:28,056 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:55:28,056 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:55:28,056 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:55:28,056 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:28,056 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:55:28,056 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:55:28,056 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:28,056 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:28,058 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:28,058 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:28,058 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:28,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:28,058 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:28,058 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:55:28,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:28,059 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:28,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:55:28,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:28,059 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:55:28,059 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:55:28,059 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:55:28,059 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:55:28,059 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:28,059 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:55:28,059 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:55:28,059 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:55:28,060 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. 2023-04-16 00:55:28,060 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. started... 2023-04-16 00:55:28,060 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. as already flushing 2023-04-16 00:55:28,060 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. 2023-04-16 00:55:28,060 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:55:28,060 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:55:28,060 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:28,060 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:55:28,060 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:55:28,061 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:28,061 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:28,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:28,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:28,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:28,061 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:28,062 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:55:28,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:28,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:28,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:28,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:28,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:28,063 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:55:28,063 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:55:28,063 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@72b337cd[Count = 0] remaining members to acquire global barrier 2023-04-16 00:55:28,063 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:28,064 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:28,064 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:28,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:28,064 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:55:28,064 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:55:28,064 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:55:28,064 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:28,064 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:55:28,072 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:28,072 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:55:28,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:28,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:28,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:28,072 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:28,072 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:55:28,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:28,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:28,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:28,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:28,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:28,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:28,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:28,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:55:28,074 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:55:28,074 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:55:28,074 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:55:28,074 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:55:28,074 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:55:28,075 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:55:28,076 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:28,076 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:28,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:28,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:28,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:28,076 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:28,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:28,076 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:28,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:28,077 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:28,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:28,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:28,078 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:28,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:28,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:28,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:28,079 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:28,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:28,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:28,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:28,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:28,080 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:28,080 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:28,080 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:28,080 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:28,081 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:55:28,081 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:28,081 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:55:28,081 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:28,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:28,081 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:28,081 DEBUG [Thread-2414] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:55:28,080 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:28,081 DEBUG [Thread-2414] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:55:28,081 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:28,081 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:28,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:28,081 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:28,081 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:28,181 DEBUG [Thread-2414] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:55:28,182 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:55:28,183 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:55:28,184 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:55:28,184 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:55:28,184 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:55:28,184 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:28,184 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:55:28,184 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:55:28,184 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:28,185 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:28,186 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:28,186 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:28,186 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:28,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:28,186 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:28,186 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:55:28,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:28,186 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:28,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:55:28,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:28,187 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:55:28,187 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:55:28,187 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:55:28,187 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:55:28,187 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:28,187 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:55:28,187 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:55:28,187 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:55:28,187 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. 2023-04-16 00:55:28,187 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. started... 2023-04-16 00:55:28,187 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. as already flushing 2023-04-16 00:55:28,187 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. 2023-04-16 00:55:28,187 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:55:28,188 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:55:28,188 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:28,188 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:55:28,188 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:55:28,190 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:28,190 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:28,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:28,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:28,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:28,190 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:28,190 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:55:28,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:28,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:28,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:28,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:28,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:28,192 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:55:28,192 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@dfb3bbe[Count = 0] remaining members to acquire global barrier 2023-04-16 00:55:28,192 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:55:28,192 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:28,193 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:28,193 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:28,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:28,193 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:28,193 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:55:28,193 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:55:28,193 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:55:28,193 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:55:28,194 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:28,194 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:55:28,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:28,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:28,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:28,194 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:28,195 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:55:28,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:28,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:28,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:28,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:28,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:28,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:28,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:28,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:55:28,197 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:55:28,197 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:55:28,197 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:55:28,197 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:55:28,197 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:55:28,197 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:55:28,198 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:28,198 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:28,198 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:28,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:28,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:28,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:28,198 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:28,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:28,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:28,199 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:28,199 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:28,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:28,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:28,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:28,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:28,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:28,200 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:28,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:28,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:28,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:28,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:28,204 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:28,204 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:28,204 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:28,204 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:55:28,204 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:55:28,204 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:55:28,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:28,204 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:28,204 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:28,204 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:28,204 DEBUG [Thread-2414] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:55:28,204 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:28,204 DEBUG [Thread-2414] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:55:28,204 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:28,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:28,204 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:28,205 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:28,205 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:28,205 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:28,213 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:28,213 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59650 deadline: 1681606588205, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:28,215 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:28,215 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59666 deadline: 1681606588207, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:28,215 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:28,215 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59642 deadline: 1681606588208, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:28,215 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:28,216 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59664 deadline: 1681606588210, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:28,305 DEBUG [Thread-2414] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:55:28,305 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:55:28,307 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:55:28,308 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:55:28,308 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:55:28,308 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:55:28,308 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:28,308 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:55:28,308 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:55:28,309 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:28,309 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:28,310 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:28,310 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:28,310 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:28,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:28,310 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:28,310 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:55:28,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:28,311 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:28,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:55:28,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:28,311 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:55:28,312 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:55:28,312 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:55:28,312 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:55:28,312 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:28,312 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:55:28,312 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:55:28,312 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:55:28,312 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. 2023-04-16 00:55:28,312 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. started... 2023-04-16 00:55:28,312 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. as already flushing 2023-04-16 00:55:28,312 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. 2023-04-16 00:55:28,312 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:55:28,312 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:55:28,312 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:28,313 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:55:28,313 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:55:28,314 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:28,314 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:28,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:28,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:28,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:28,314 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:28,314 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:55:28,315 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=331 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/.tmp/B/41512abe23d34f36b66e488ca81523d1 2023-04-16 00:55:28,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:28,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:28,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:28,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:28,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:28,317 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:55:28,317 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@12159eff[Count = 0] remaining members to acquire global barrier 2023-04-16 00:55:28,317 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:55:28,317 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:28,319 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:28,319 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:28,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:28,319 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:55:28,319 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:55:28,319 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:28,319 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:55:28,319 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:55:28,321 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:55:28,322 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:28,322 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:55:28,322 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:28,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:28,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:28,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:28,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:28,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:28,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:28,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:28,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:28,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:28,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:28,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:55:28,325 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:55:28,325 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:55:28,325 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:55:28,325 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:55:28,325 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:55:28,325 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:55:28,326 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=331 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/.tmp/C/667ca7c094594c9eb7a4a5fbb6614e26 2023-04-16 00:55:28,328 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:28,328 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:28,328 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:28,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:28,328 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:28,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:28,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:28,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:28,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:28,329 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:28,329 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:28,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:28,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:28,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:28,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:28,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:28,331 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:28,331 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/.tmp/A/a1de6d30d39b472f8f371b2fe15e3c2a as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/A/a1de6d30d39b472f8f371b2fe15e3c2a 2023-04-16 00:55:28,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:28,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:28,335 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/A/a1de6d30d39b472f8f371b2fe15e3c2a, entries=200, sequenceid=331, filesize=32.3 K 2023-04-16 00:55:28,335 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/.tmp/B/41512abe23d34f36b66e488ca81523d1 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/B/41512abe23d34f36b66e488ca81523d1 2023-04-16 00:55:28,336 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:28,336 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:28,336 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:55:28,336 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:55:28,336 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:28,336 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:55:28,336 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:28,337 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:28,336 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:28,337 DEBUG [Thread-2414] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:55:28,336 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:28,337 DEBUG [Thread-2414] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:55:28,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:28,337 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:28,337 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:28,337 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:28,337 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:28,337 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:28,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:28,340 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/B/41512abe23d34f36b66e488ca81523d1, entries=150, sequenceid=331, filesize=11.9 K 2023-04-16 00:55:28,340 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/.tmp/C/667ca7c094594c9eb7a4a5fbb6614e26 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/C/667ca7c094594c9eb7a4a5fbb6614e26 2023-04-16 00:55:28,345 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/C/667ca7c094594c9eb7a4a5fbb6614e26, entries=150, sequenceid=331, filesize=11.9 K 2023-04-16 00:55:28,346 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for ee2f682ad6dd0fe8e91c021d54d3bdc5 in 922ms, sequenceid=331, compaction requested=true 2023-04-16 00:55:28,346 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for ee2f682ad6dd0fe8e91c021d54d3bdc5: 2023-04-16 00:55:28,346 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 00:55:28,346 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-04-16 00:55:28,346 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-04-16 00:55:28,346 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 00:55:28,346 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 00:55:28,350 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 7 files of size 84724 starting at candidate #0 after considering 15 permutations with 15 in ratio 2023-04-16 00:55:28,350 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HStore(1912): ee2f682ad6dd0fe8e91c021d54d3bdc5/B is initiating minor compaction (all files) 2023-04-16 00:55:28,350 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HRegion(2259): Starting compaction of ee2f682ad6dd0fe8e91c021d54d3bdc5/B in TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. 2023-04-16 00:55:28,350 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/B/2b0f80526d1b4839b99cb342be941119, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/B/054b59a0405246e8b8017056a77f2352, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/B/7d50407fb9fe4c1e902f8ded55f85a01, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/B/2c02ceba33224305bed1757852beaca3, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/B/67320ea88b954a078c0608b9799b3e08, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/B/578d5bb5951a4745b077be16e0997cfa, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/B/41512abe23d34f36b66e488ca81523d1] into tmpdir=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/.tmp, totalSize=82.7 K 2023-04-16 00:55:28,351 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 7 files of size 195774 starting at candidate #0 after considering 15 permutations with 15 in ratio 2023-04-16 00:55:28,351 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HStore(1912): ee2f682ad6dd0fe8e91c021d54d3bdc5/A is initiating minor compaction (all files) 2023-04-16 00:55:28,351 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of ee2f682ad6dd0fe8e91c021d54d3bdc5/A in TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. 2023-04-16 00:55:28,351 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/A/2071d651f31a4115bfe4039c27575f60, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/A/932b3720baa94283b6e6b897b5ac522f, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/A/182cec2bc8564e7488bff17b333a205f, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/A/678fac82f9f74535baae96803b8563de, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/A/fcb8dc602cda460989697a888a026465, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/A/8ac9840dd9f84421b70650c911dd55c3, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/A/a1de6d30d39b472f8f371b2fe15e3c2a] into tmpdir=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/.tmp, totalSize=191.2 K 2023-04-16 00:55:28,351 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting 2b0f80526d1b4839b99cb342be941119, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=104, earliestPutTs=1681606521569 2023-04-16 00:55:28,352 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.Compactor(207): Compacting 2071d651f31a4115bfe4039c27575f60, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=104, earliestPutTs=1681606521569 2023-04-16 00:55:28,353 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.Compactor(207): Compacting 932b3720baa94283b6e6b897b5ac522f, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=140, earliestPutTs=1681606521759 2023-04-16 00:55:28,353 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting 054b59a0405246e8b8017056a77f2352, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=140, earliestPutTs=1681606521759 2023-04-16 00:55:28,354 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.Compactor(207): Compacting 182cec2bc8564e7488bff17b333a205f, keycount=200, bloomtype=ROW, size=32.1 K, encoding=NONE, compression=NONE, seqNum=147, earliestPutTs=1681606522963 2023-04-16 00:55:28,354 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting 7d50407fb9fe4c1e902f8ded55f85a01, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=147, earliestPutTs=1681606522963 2023-04-16 00:55:28,354 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting 2c02ceba33224305bed1757852beaca3, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=256, earliestPutTs=1681606523272 2023-04-16 00:55:28,355 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.Compactor(207): Compacting 678fac82f9f74535baae96803b8563de, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=256, earliestPutTs=1681606523272 2023-04-16 00:55:28,355 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting 67320ea88b954a078c0608b9799b3e08, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=264, earliestPutTs=1681606524420 2023-04-16 00:55:28,355 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.Compactor(207): Compacting fcb8dc602cda460989697a888a026465, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=264, earliestPutTs=1681606524420 2023-04-16 00:55:28,355 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting 578d5bb5951a4745b077be16e0997cfa, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=318, earliestPutTs=1681606525189 2023-04-16 00:55:28,355 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.Compactor(207): Compacting 8ac9840dd9f84421b70650c911dd55c3, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=318, earliestPutTs=1681606525189 2023-04-16 00:55:28,356 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting 41512abe23d34f36b66e488ca81523d1, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=331, earliestPutTs=1681606527368 2023-04-16 00:55:28,361 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.Compactor(207): Compacting a1de6d30d39b472f8f371b2fe15e3c2a, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=331, earliestPutTs=1681606527368 2023-04-16 00:55:28,383 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] throttle.PressureAwareThroughputController(145): ee2f682ad6dd0fe8e91c021d54d3bdc5#B#compaction#351 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-16 00:55:28,399 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] throttle.PressureAwareThroughputController(145): ee2f682ad6dd0fe8e91c021d54d3bdc5#A#compaction#352 average throughput is 1.41 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-16 00:55:28,437 DEBUG [Thread-2414] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:55:28,437 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:55:28,439 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:55:28,440 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:55:28,440 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:55:28,440 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:55:28,440 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:28,440 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:55:28,440 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:55:28,440 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:28,440 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:28,443 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:28,443 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:28,443 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:28,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:28,443 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:28,443 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:55:28,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:28,444 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:28,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:55:28,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:28,444 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:55:28,444 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:55:28,444 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:55:28,444 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:55:28,444 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:28,444 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:55:28,444 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:55:28,444 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:55:28,445 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. 2023-04-16 00:55:28,445 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. started... 2023-04-16 00:55:28,445 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing ee2f682ad6dd0fe8e91c021d54d3bdc5 3/3 column families, dataSize=134.18 KB heapSize=375.61 KB 2023-04-16 00:55:28,445 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ee2f682ad6dd0fe8e91c021d54d3bdc5, store=A 2023-04-16 00:55:28,445 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 00:55:28,445 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ee2f682ad6dd0fe8e91c021d54d3bdc5, store=B 2023-04-16 00:55:28,445 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 00:55:28,445 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ee2f682ad6dd0fe8e91c021d54d3bdc5, store=C 2023-04-16 00:55:28,445 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 00:55:28,468 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230416a0bf3deb697a4343b03479b66f1b327c to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230416a0bf3deb697a4343b03479b66f1b327c 2023-04-16 00:55:28,474 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=366, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/.tmp/A/71c120ce99bc4fe7ae656fd0cd4f9285 2023-04-16 00:55:28,490 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=366 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/.tmp/B/e21f9ae5cfbf4cbeb56266c98cbfab59 2023-04-16 00:55:28,504 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=366 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/.tmp/C/3294108abd13461d82dfb6cd003b584b 2023-04-16 00:55:28,508 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/.tmp/A/71c120ce99bc4fe7ae656fd0cd4f9285 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/A/71c120ce99bc4fe7ae656fd0cd4f9285 2023-04-16 00:55:28,511 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/A/71c120ce99bc4fe7ae656fd0cd4f9285, entries=150, sequenceid=366, filesize=25.4 K 2023-04-16 00:55:28,512 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/.tmp/B/e21f9ae5cfbf4cbeb56266c98cbfab59 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/B/e21f9ae5cfbf4cbeb56266c98cbfab59 2023-04-16 00:55:28,515 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/B/e21f9ae5cfbf4cbeb56266c98cbfab59, entries=150, sequenceid=366, filesize=11.9 K 2023-04-16 00:55:28,516 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/.tmp/C/3294108abd13461d82dfb6cd003b584b as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/C/3294108abd13461d82dfb6cd003b584b 2023-04-16 00:55:28,527 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/C/3294108abd13461d82dfb6cd003b584b, entries=150, sequenceid=366, filesize=11.9 K 2023-04-16 00:55:28,527 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~375.56 KB/384576, currentSize=0 B/0 for ee2f682ad6dd0fe8e91c021d54d3bdc5 in 82ms, sequenceid=366, compaction requested=true 2023-04-16 00:55:28,528 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for ee2f682ad6dd0fe8e91c021d54d3bdc5: 2023-04-16 00:55:28,528 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. 2023-04-16 00:55:28,528 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:55:28,528 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:55:28,528 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:28,528 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:55:28,528 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:55:28,530 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:28,530 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:28,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:28,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:28,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:28,530 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:28,530 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:55:28,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:28,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:28,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:28,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:28,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:28,532 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:55:28,532 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@594297ca[Count = 0] remaining members to acquire global barrier 2023-04-16 00:55:28,532 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:55:28,532 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:28,533 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:28,533 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:28,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:28,534 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:55:28,534 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:55:28,534 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:55:28,534 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:28,534 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:55:28,535 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:55:28,535 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:28,535 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:28,535 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:55:28,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:28,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:28,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:28,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:28,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:28,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:28,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:28,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:28,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:28,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:28,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:55:28,538 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:55:28,538 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:55:28,538 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:55:28,538 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:55:28,538 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:55:28,538 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:55:28,541 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:28,541 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:28,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:28,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:28,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:28,541 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:28,541 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:28,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:28,542 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:28,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:28,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:28,542 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:28,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:28,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:28,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:28,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:28,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:28,544 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:28,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:28,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:28,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:28,546 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:28,547 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:55:28,547 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:28,547 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:28,547 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:28,547 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:28,547 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:55:28,547 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:28,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:28,547 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:28,547 DEBUG [Thread-2414] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:55:28,547 DEBUG [Thread-2414] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:55:28,547 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:28,547 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:28,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:28,547 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:28,547 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:28,548 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:28,647 DEBUG [Thread-2414] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:55:28,648 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:55:28,649 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:55:28,651 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:55:28,651 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:55:28,651 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:55:28,651 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:28,651 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:55:28,651 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:55:28,652 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:28,652 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:28,653 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:28,653 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:28,653 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:28,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:28,653 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:28,653 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:55:28,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:28,654 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:28,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:55:28,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:28,654 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:55:28,655 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:55:28,655 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:55:28,655 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:55:28,655 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:28,655 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:55:28,655 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:55:28,655 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:55:28,655 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. 2023-04-16 00:55:28,655 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. started... 2023-04-16 00:55:28,655 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for ee2f682ad6dd0fe8e91c021d54d3bdc5: 2023-04-16 00:55:28,655 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. 2023-04-16 00:55:28,655 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:55:28,655 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:55:28,655 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:28,655 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:55:28,655 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:55:28,657 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:28,657 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:28,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:28,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:28,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:28,657 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:28,657 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:55:28,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:28,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:28,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:28,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:28,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:28,659 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:55:28,659 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:55:28,659 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3224de70[Count = 0] remaining members to acquire global barrier 2023-04-16 00:55:28,659 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:28,661 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:28,661 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:28,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:28,662 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:55:28,662 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:55:28,662 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:55:28,662 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:28,662 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:55:28,663 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:28,663 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:55:28,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:28,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:28,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:28,663 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:28,663 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:55:28,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:28,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:28,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:28,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:28,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:28,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:28,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:28,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:55:28,667 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:55:28,667 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:55:28,667 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:55:28,667 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:55:28,667 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:55:28,667 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:55:28,668 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:28,668 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:28,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:28,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:28,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:28,668 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:28,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:28,668 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:28,669 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:28,669 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:28,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:28,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:28,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:28,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:28,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:28,670 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:28,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:28,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:28,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:28,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:28,672 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:28,672 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:28,672 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:28,672 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:28,672 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:28,672 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:28,672 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:28,672 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:28,672 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:28,672 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:28,672 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:55:28,672 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:55:28,672 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:28,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:28,672 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:55:28,673 DEBUG [Thread-2414] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:55:28,673 DEBUG [Thread-2414] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:55:28,673 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:28,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:28,773 DEBUG [Thread-2414] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:55:28,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:55:28,775 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:55:28,776 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:55:28,776 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:55:28,776 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:55:28,776 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:28,776 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:55:28,776 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:55:28,777 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:28,777 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:28,779 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:28,779 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:28,779 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:28,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:28,780 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:28,780 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:55:28,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:28,780 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:28,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:55:28,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:28,780 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:55:28,781 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:55:28,781 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:55:28,781 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:55:28,781 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:28,781 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:55:28,781 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:55:28,781 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:55:28,781 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. 2023-04-16 00:55:28,781 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. started... 2023-04-16 00:55:28,781 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing ee2f682ad6dd0fe8e91c021d54d3bdc5 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-04-16 00:55:28,781 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ee2f682ad6dd0fe8e91c021d54d3bdc5, store=A 2023-04-16 00:55:28,781 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:55:28,781 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ee2f682ad6dd0fe8e91c021d54d3bdc5, store=B 2023-04-16 00:55:28,781 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:55:28,781 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ee2f682ad6dd0fe8e91c021d54d3bdc5, store=C 2023-04-16 00:55:28,781 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:55:28,798 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/.tmp/B/e5729e88d4db4396a45a9ba5a4709ecf as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/B/e5729e88d4db4396a45a9ba5a4709ecf 2023-04-16 00:55:28,799 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304165efeb500254e48aeb84708afe2a65db2 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304165efeb500254e48aeb84708afe2a65db2 2023-04-16 00:55:28,806 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in ee2f682ad6dd0fe8e91c021d54d3bdc5/B of ee2f682ad6dd0fe8e91c021d54d3bdc5 into e5729e88d4db4396a45a9ba5a4709ecf(size=12.3 K), total size for store is 24.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 00:55:28,808 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for ee2f682ad6dd0fe8e91c021d54d3bdc5: 2023-04-16 00:55:28,808 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5., storeName=ee2f682ad6dd0fe8e91c021d54d3bdc5/B, priority=9, startTime=1681606528346; duration=0sec 2023-04-16 00:55:28,809 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 00:55:28,809 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-04-16 00:55:28,810 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=374, memsize=8.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/.tmp/A/418be8f08cb8465fbe40a26df541c06b 2023-04-16 00:55:28,811 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 7 files of size 85058 starting at candidate #0 after considering 15 permutations with 15 in ratio 2023-04-16 00:55:28,811 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HStore(1912): ee2f682ad6dd0fe8e91c021d54d3bdc5/C is initiating minor compaction (all files) 2023-04-16 00:55:28,811 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HRegion(2259): Starting compaction of ee2f682ad6dd0fe8e91c021d54d3bdc5/C in TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. 2023-04-16 00:55:28,811 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/C/5cd6c75d70c8467c9df564e03dd709f2, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/C/f78762c94569489a86c3607be4af2a47, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/C/29c995d0bad14ddeb5ac50be29be5566, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/C/0065cc7bde58460e98227dcbec7c11cc, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/C/89572d68c50148d89d56409b4eab3f9b, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/C/667ca7c094594c9eb7a4a5fbb6614e26, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/C/3294108abd13461d82dfb6cd003b584b] into tmpdir=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/.tmp, totalSize=83.1 K 2023-04-16 00:55:28,812 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting 5cd6c75d70c8467c9df564e03dd709f2, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=140, earliestPutTs=1681606521759 2023-04-16 00:55:28,812 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting f78762c94569489a86c3607be4af2a47, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=147, earliestPutTs=1681606522963 2023-04-16 00:55:28,813 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting 29c995d0bad14ddeb5ac50be29be5566, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=256, earliestPutTs=1681606523272 2023-04-16 00:55:28,813 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting 0065cc7bde58460e98227dcbec7c11cc, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=264, earliestPutTs=1681606524420 2023-04-16 00:55:28,813 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting 89572d68c50148d89d56409b4eab3f9b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=318, earliestPutTs=1681606525189 2023-04-16 00:55:28,814 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting 667ca7c094594c9eb7a4a5fbb6614e26, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=331, earliestPutTs=1681606527368 2023-04-16 00:55:28,814 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting 3294108abd13461d82dfb6cd003b584b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=366, earliestPutTs=1681606527556 2023-04-16 00:55:28,821 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=374 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/.tmp/B/0fdda349dad24a0982930e05a8ce1053 2023-04-16 00:55:28,826 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(9158): Flush requested on ee2f682ad6dd0fe8e91c021d54d3bdc5 2023-04-16 00:55:28,826 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. as already flushing 2023-04-16 00:55:28,828 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] throttle.PressureAwareThroughputController(145): ee2f682ad6dd0fe8e91c021d54d3bdc5#C#compaction#358 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-16 00:55:28,839 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=374 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/.tmp/C/157ff7be7222483aa909be17f0f6304b 2023-04-16 00:55:28,842 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/.tmp/A/418be8f08cb8465fbe40a26df541c06b as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/A/418be8f08cb8465fbe40a26df541c06b 2023-04-16 00:55:28,846 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/A/418be8f08cb8465fbe40a26df541c06b, entries=200, sequenceid=374, filesize=32.3 K 2023-04-16 00:55:28,846 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/.tmp/B/0fdda349dad24a0982930e05a8ce1053 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/B/0fdda349dad24a0982930e05a8ce1053 2023-04-16 00:55:28,849 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/B/0fdda349dad24a0982930e05a8ce1053, entries=200, sequenceid=374, filesize=14.3 K 2023-04-16 00:55:28,850 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/.tmp/C/157ff7be7222483aa909be17f0f6304b as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/C/157ff7be7222483aa909be17f0f6304b 2023-04-16 00:55:28,856 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/C/157ff7be7222483aa909be17f0f6304b, entries=100, sequenceid=374, filesize=9.5 K 2023-04-16 00:55:28,857 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=53.67 KB/54960 for ee2f682ad6dd0fe8e91c021d54d3bdc5 in 76ms, sequenceid=374, compaction requested=true 2023-04-16 00:55:28,857 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for ee2f682ad6dd0fe8e91c021d54d3bdc5: 2023-04-16 00:55:28,857 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. 2023-04-16 00:55:28,857 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:55:28,857 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:55:28,857 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:28,857 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:55:28,857 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:55:28,859 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:28,859 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:28,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:28,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:28,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:28,859 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(9158): Flush requested on ee2f682ad6dd0fe8e91c021d54d3bdc5 2023-04-16 00:55:28,859 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing ee2f682ad6dd0fe8e91c021d54d3bdc5 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-16 00:55:28,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:28,859 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:28,860 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ee2f682ad6dd0fe8e91c021d54d3bdc5, store=A 2023-04-16 00:55:28,860 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:55:28,860 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ee2f682ad6dd0fe8e91c021d54d3bdc5, store=B 2023-04-16 00:55:28,860 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:55:28,860 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:55:28,860 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ee2f682ad6dd0fe8e91c021d54d3bdc5, store=C 2023-04-16 00:55:28,860 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:55:28,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:28,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:28,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:28,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:28,861 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:55:28,861 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@66562541[Count = 0] remaining members to acquire global barrier 2023-04-16 00:55:28,861 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:55:28,861 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:28,863 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:28,863 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:28,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:28,863 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:28,863 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:55:28,863 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:55:28,863 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:55:28,863 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:55:28,864 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:28,864 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:55:28,864 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:28,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:28,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:28,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:28,864 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:55:28,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:28,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:28,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:28,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:28,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:28,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:28,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:28,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:55:28,867 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:55:28,867 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:55:28,867 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:55:28,867 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:55:28,867 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:55:28,867 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:55:28,869 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:28,869 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:28,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:28,869 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:28,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:28,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:28,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:28,869 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:28,870 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:28,870 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:28,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:28,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:28,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:28,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:28,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:28,871 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:28,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:28,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:28,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:28,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:28,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:28,874 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:28,874 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:28,874 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:28,874 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:28,874 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:28,874 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:55:28,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:28,874 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:28,875 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:28,874 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:55:28,875 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:28,875 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:28,875 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:28,875 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:28,875 DEBUG [Thread-2414] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:55:28,875 DEBUG [Thread-2414] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:55:28,875 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:28,875 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:55:28,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:28,878 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023041636e9ea1f89ab49ca81c501edaf5ca24c to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041636e9ea1f89ab49ca81c501edaf5ca24c 2023-04-16 00:55:28,883 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=387, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/.tmp/A/08a334d7fa7742b29e578e57fd47127b 2023-04-16 00:55:28,924 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:28,924 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59664 deadline: 1681606588911, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:28,931 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:28,932 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59642 deadline: 1681606588920, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:28,938 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:28,938 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59650 deadline: 1681606588923, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:28,938 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:28,938 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59666 deadline: 1681606588924, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:28,975 DEBUG [Thread-2414] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:55:28,975 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:55:28,977 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:55:28,978 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:55:28,978 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:55:28,978 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:55:28,978 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:28,978 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:55:28,978 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:55:28,978 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:28,979 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:28,979 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:28,979 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:28,980 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:28,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:28,980 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:28,980 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:55:28,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:28,980 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:28,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:55:28,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:28,981 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:55:28,981 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:55:28,981 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:55:28,981 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:55:28,981 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:28,981 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:55:28,981 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:55:28,981 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:55:28,981 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. 2023-04-16 00:55:28,981 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. started... 2023-04-16 00:55:28,981 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. as already flushing 2023-04-16 00:55:28,981 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. 2023-04-16 00:55:28,981 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:55:28,981 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:55:28,981 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:28,981 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:55:28,981 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:55:28,986 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:28,986 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:28,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:28,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:28,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:28,986 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:28,986 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:55:28,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:28,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:28,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:28,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:28,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:28,987 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:55:28,987 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@580abd12[Count = 0] remaining members to acquire global barrier 2023-04-16 00:55:28,987 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:55:28,987 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:28,988 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:28,988 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:28,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:28,988 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:55:28,988 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:55:28,988 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:55:28,988 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:28,989 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:55:28,990 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:55:28,990 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:28,990 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:28,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:28,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:28,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:28,990 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:55:28,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:28,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:28,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:28,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:28,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:28,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:28,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:28,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:55:28,992 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:55:28,992 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:55:28,992 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:55:28,992 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:55:28,992 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:55:28,992 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:55:28,994 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:28,994 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:28,994 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:28,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:28,994 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:28,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:28,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:28,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:28,994 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:28,995 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:28,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:28,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:28,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:28,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:28,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:28,996 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:28,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:28,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:28,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:28,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:28,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:28,997 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:28,997 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:28,997 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:28,997 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:28,998 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:28,998 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:28,998 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:28,998 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:28,998 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:28,998 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:55:28,998 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:55:28,998 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:28,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:28,998 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:28,998 DEBUG [Thread-2414] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:55:28,998 DEBUG [Thread-2414] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:55:28,998 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:28,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:28,999 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:55:29,040 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:29,040 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59664 deadline: 1681606589025, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:29,041 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:29,041 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59642 deadline: 1681606589033, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:29,047 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:29,047 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 159 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59650 deadline: 1681606589039, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:29,047 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:29,048 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59666 deadline: 1681606589039, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:29,098 DEBUG [Thread-2414] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:55:29,099 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:55:29,101 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:55:29,102 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:55:29,102 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:55:29,102 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:55:29,102 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:29,102 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:55:29,102 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:55:29,102 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:29,102 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:29,103 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:29,104 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:29,104 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:55:29,104 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:29,104 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:29,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:29,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:29,105 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:29,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:55:29,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:29,105 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:55:29,106 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:55:29,106 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:55:29,106 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:55:29,106 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:29,106 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:55:29,106 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:55:29,106 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:55:29,106 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. 2023-04-16 00:55:29,106 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. started... 2023-04-16 00:55:29,107 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. as already flushing 2023-04-16 00:55:29,107 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. 2023-04-16 00:55:29,107 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:55:29,107 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:55:29,107 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:29,107 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:55:29,107 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:55:29,108 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:29,108 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:29,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:29,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:29,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:29,109 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:29,109 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:55:29,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:29,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:29,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:29,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:29,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:29,110 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:55:29,110 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:55:29,110 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@153b8d1e[Count = 0] remaining members to acquire global barrier 2023-04-16 00:55:29,110 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:29,112 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:29,112 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:29,112 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:55:29,112 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:29,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:29,112 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:55:29,112 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:55:29,112 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:55:29,115 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:55:29,115 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:29,115 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:29,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:29,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:29,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:29,115 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:55:29,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:29,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:29,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:29,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:29,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:29,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:29,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:29,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:55:29,119 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:55:29,119 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:55:29,119 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:55:29,119 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:55:29,119 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:55:29,119 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:55:29,120 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:29,120 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:29,120 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:29,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:29,120 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:29,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:29,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:29,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:29,121 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:29,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:29,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:29,121 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:29,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:29,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:29,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:29,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:29,122 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:29,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:29,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:29,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:29,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:29,124 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:29,124 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:29,124 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:29,124 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:29,124 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:29,124 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:55:29,124 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:55:29,125 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:55:29,124 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:29,125 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:29,125 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:29,125 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:29,124 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:29,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:29,124 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:29,125 DEBUG [Thread-2414] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:55:29,125 DEBUG [Thread-2414] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:55:29,126 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:29,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:29,225 DEBUG [Thread-2414] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:55:29,226 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:55:29,227 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:55:29,228 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:55:29,228 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:55:29,228 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:55:29,228 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:29,228 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:55:29,228 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:55:29,229 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:29,229 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:29,230 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:29,230 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:29,230 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:29,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:29,230 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:29,230 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:55:29,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:29,230 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:29,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:55:29,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:29,231 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:55:29,231 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:55:29,231 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:55:29,231 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:55:29,231 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:29,231 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:55:29,231 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:55:29,231 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:55:29,231 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. 2023-04-16 00:55:29,231 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. started... 2023-04-16 00:55:29,232 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. as already flushing 2023-04-16 00:55:29,232 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. 2023-04-16 00:55:29,232 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:55:29,232 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:55:29,232 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:29,232 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:55:29,232 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:55:29,233 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:29,233 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:29,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:29,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:29,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:29,234 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:29,234 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:55:29,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:29,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:29,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:29,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:29,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:29,235 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:55:29,235 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:55:29,235 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1781697[Count = 0] remaining members to acquire global barrier 2023-04-16 00:55:29,235 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:29,243 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:29,243 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:29,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:29,243 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:55:29,243 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:55:29,243 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:55:29,243 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:29,243 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:55:29,244 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/.tmp/A/fc876d5198cf459d8a9c1daa73d0369c as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/A/fc876d5198cf459d8a9c1daa73d0369c 2023-04-16 00:55:29,244 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:55:29,244 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:29,244 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:29,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:29,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:29,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:29,244 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:55:29,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:29,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:29,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:29,245 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/.tmp/C/a59c947f064844b0b361c5e65f5f3869 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/C/a59c947f064844b0b361c5e65f5f3869 2023-04-16 00:55:29,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:29,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:29,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:29,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:29,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:55:29,247 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:55:29,247 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:55:29,247 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:55:29,247 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:55:29,247 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:55:29,247 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:55:29,249 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in ee2f682ad6dd0fe8e91c021d54d3bdc5/A of ee2f682ad6dd0fe8e91c021d54d3bdc5 into fc876d5198cf459d8a9c1daa73d0369c(size=25.8 K), total size for store is 83.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 00:55:29,249 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for ee2f682ad6dd0fe8e91c021d54d3bdc5: 2023-04-16 00:55:29,249 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:29,250 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59642 deadline: 1681606589242, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:29,250 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:29,250 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:29,250 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:29,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:29,249 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5., storeName=ee2f682ad6dd0fe8e91c021d54d3bdc5/A, priority=9, startTime=1681606528346; duration=0sec 2023-04-16 00:55:29,250 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 00:55:29,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:29,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:29,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:29,250 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:29,250 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in ee2f682ad6dd0fe8e91c021d54d3bdc5/C of ee2f682ad6dd0fe8e91c021d54d3bdc5 into a59c947f064844b0b361c5e65f5f3869(size=12.3 K), total size for store is 21.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 00:55:29,250 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for ee2f682ad6dd0fe8e91c021d54d3bdc5: 2023-04-16 00:55:29,250 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5., storeName=ee2f682ad6dd0fe8e91c021d54d3bdc5/C, priority=9, startTime=1681606528346; duration=0sec 2023-04-16 00:55:29,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:29,251 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:29,251 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 00:55:29,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:29,251 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:29,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:29,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:29,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:29,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:29,252 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:29,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:29,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:29,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:29,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:29,254 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:29,254 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:29,254 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:29,254 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:29,254 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:29,254 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:55:29,254 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:55:29,254 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:29,254 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:29,254 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:29,254 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:29,254 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:29,254 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:29,254 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:29,254 DEBUG [Thread-2414] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:55:29,255 DEBUG [Thread-2414] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:55:29,254 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59664 deadline: 1681606589242, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:29,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:29,255 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:29,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:29,257 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:29,257 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59666 deadline: 1681606589249, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:29,257 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:29,257 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 161 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59650 deadline: 1681606589249, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:29,292 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=387 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/.tmp/B/7f59ab69e3be495aa277833d85b29246 2023-04-16 00:55:29,310 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=387 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/.tmp/C/ab710e19efc841b298189bda8852f1ee 2023-04-16 00:55:29,316 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/.tmp/A/08a334d7fa7742b29e578e57fd47127b as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/A/08a334d7fa7742b29e578e57fd47127b 2023-04-16 00:55:29,320 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/A/08a334d7fa7742b29e578e57fd47127b, entries=450, sequenceid=387, filesize=66.7 K 2023-04-16 00:55:29,321 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/.tmp/B/7f59ab69e3be495aa277833d85b29246 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/B/7f59ab69e3be495aa277833d85b29246 2023-04-16 00:55:29,325 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/B/7f59ab69e3be495aa277833d85b29246, entries=450, sequenceid=387, filesize=26.2 K 2023-04-16 00:55:29,325 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/.tmp/C/ab710e19efc841b298189bda8852f1ee as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/C/ab710e19efc841b298189bda8852f1ee 2023-04-16 00:55:29,328 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/C/ab710e19efc841b298189bda8852f1ee, entries=150, sequenceid=387, filesize=11.9 K 2023-04-16 00:55:29,329 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for ee2f682ad6dd0fe8e91c021d54d3bdc5 in 470ms, sequenceid=387, compaction requested=true 2023-04-16 00:55:29,329 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for ee2f682ad6dd0fe8e91c021d54d3bdc5: 2023-04-16 00:55:29,329 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 00:55:29,329 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-16 00:55:29,329 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-16 00:55:29,329 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-16 00:55:29,330 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-16 00:55:29,331 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 153878 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-16 00:55:29,331 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HStore(1912): ee2f682ad6dd0fe8e91c021d54d3bdc5/A is initiating minor compaction (all files) 2023-04-16 00:55:29,331 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of ee2f682ad6dd0fe8e91c021d54d3bdc5/A in TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. 2023-04-16 00:55:29,331 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/A/fc876d5198cf459d8a9c1daa73d0369c, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/A/71c120ce99bc4fe7ae656fd0cd4f9285, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/A/418be8f08cb8465fbe40a26df541c06b, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/A/08a334d7fa7742b29e578e57fd47127b] into tmpdir=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/.tmp, totalSize=150.3 K 2023-04-16 00:55:29,331 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 66234 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-16 00:55:29,331 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.Compactor(207): Compacting fc876d5198cf459d8a9c1daa73d0369c, keycount=150, bloomtype=ROW, size=25.8 K, encoding=NONE, compression=NONE, seqNum=331, earliestPutTs=1681606527368 2023-04-16 00:55:29,332 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HStore(1912): ee2f682ad6dd0fe8e91c021d54d3bdc5/B is initiating minor compaction (all files) 2023-04-16 00:55:29,332 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HRegion(2259): Starting compaction of ee2f682ad6dd0fe8e91c021d54d3bdc5/B in TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. 2023-04-16 00:55:29,332 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/B/e5729e88d4db4396a45a9ba5a4709ecf, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/B/e21f9ae5cfbf4cbeb56266c98cbfab59, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/B/0fdda349dad24a0982930e05a8ce1053, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/B/7f59ab69e3be495aa277833d85b29246] into tmpdir=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/.tmp, totalSize=64.7 K 2023-04-16 00:55:29,333 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.Compactor(207): Compacting 71c120ce99bc4fe7ae656fd0cd4f9285, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=366, earliestPutTs=1681606527556 2023-04-16 00:55:29,333 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting e5729e88d4db4396a45a9ba5a4709ecf, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=331, earliestPutTs=1681606527368 2023-04-16 00:55:29,333 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.Compactor(207): Compacting 418be8f08cb8465fbe40a26df541c06b, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=374, earliestPutTs=1681606528763 2023-04-16 00:55:29,334 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting e21f9ae5cfbf4cbeb56266c98cbfab59, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=366, earliestPutTs=1681606527556 2023-04-16 00:55:29,334 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.Compactor(207): Compacting 08a334d7fa7742b29e578e57fd47127b, keycount=450, bloomtype=ROW, size=66.7 K, encoding=NONE, compression=NONE, seqNum=387, earliestPutTs=1681606528823 2023-04-16 00:55:29,334 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting 0fdda349dad24a0982930e05a8ce1053, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=374, earliestPutTs=1681606528763 2023-04-16 00:55:29,335 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting 7f59ab69e3be495aa277833d85b29246, keycount=450, bloomtype=ROW, size=26.2 K, encoding=NONE, compression=NONE, seqNum=387, earliestPutTs=1681606528823 2023-04-16 00:55:29,346 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] throttle.PressureAwareThroughputController(145): ee2f682ad6dd0fe8e91c021d54d3bdc5#B#compaction#364 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-16 00:55:29,355 DEBUG [Thread-2414] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:55:29,355 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] throttle.PressureAwareThroughputController(145): ee2f682ad6dd0fe8e91c021d54d3bdc5#A#compaction#363 average throughput is 1.79 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-16 00:55:29,359 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:55:29,361 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:55:29,365 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:55:29,365 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:55:29,365 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:55:29,365 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:29,365 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:55:29,365 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:55:29,365 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:29,365 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:29,367 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:29,367 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:29,367 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:29,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:29,368 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:29,368 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:55:29,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:29,368 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:29,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:55:29,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:29,369 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:55:29,376 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:55:29,376 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:55:29,376 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:55:29,376 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:29,376 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:55:29,376 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:55:29,376 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:55:29,376 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. 2023-04-16 00:55:29,376 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. started... 2023-04-16 00:55:29,376 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing ee2f682ad6dd0fe8e91c021d54d3bdc5 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-16 00:55:29,377 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ee2f682ad6dd0fe8e91c021d54d3bdc5, store=A 2023-04-16 00:55:29,377 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:55:29,377 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ee2f682ad6dd0fe8e91c021d54d3bdc5, store=B 2023-04-16 00:55:29,377 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:55:29,377 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ee2f682ad6dd0fe8e91c021d54d3bdc5, store=C 2023-04-16 00:55:29,377 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:55:29,556 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(9158): Flush requested on ee2f682ad6dd0fe8e91c021d54d3bdc5 2023-04-16 00:55:29,557 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. as already flushing 2023-04-16 00:55:29,574 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:29,574 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59666 deadline: 1681606589565, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:29,574 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:29,574 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59642 deadline: 1681606589566, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:29,580 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:29,580 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59650 deadline: 1681606589572, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:29,585 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:29,585 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 159 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59664 deadline: 1681606589574, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:29,679 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:29,679 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59666 deadline: 1681606589675, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:29,680 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:29,680 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 162 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59642 deadline: 1681606589675, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:29,686 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:29,686 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59650 deadline: 1681606589681, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:29,690 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:29,690 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 161 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59664 deadline: 1681606589685, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:29,760 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/.tmp/B/529f111b0dd2430387ad55fd414e7016 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/B/529f111b0dd2430387ad55fd414e7016 2023-04-16 00:55:29,764 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in ee2f682ad6dd0fe8e91c021d54d3bdc5/B of ee2f682ad6dd0fe8e91c021d54d3bdc5 into 529f111b0dd2430387ad55fd414e7016(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 00:55:29,764 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for ee2f682ad6dd0fe8e91c021d54d3bdc5: 2023-04-16 00:55:29,764 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5., storeName=ee2f682ad6dd0fe8e91c021d54d3bdc5/B, priority=12, startTime=1681606529329; duration=0sec 2023-04-16 00:55:29,764 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 00:55:29,764 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-16 00:55:29,765 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 34558 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-16 00:55:29,765 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HStore(1912): ee2f682ad6dd0fe8e91c021d54d3bdc5/C is initiating minor compaction (all files) 2023-04-16 00:55:29,765 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HRegion(2259): Starting compaction of ee2f682ad6dd0fe8e91c021d54d3bdc5/C in TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. 2023-04-16 00:55:29,765 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/C/a59c947f064844b0b361c5e65f5f3869, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/C/157ff7be7222483aa909be17f0f6304b, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/C/ab710e19efc841b298189bda8852f1ee] into tmpdir=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/.tmp, totalSize=33.7 K 2023-04-16 00:55:29,766 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting a59c947f064844b0b361c5e65f5f3869, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=366, earliestPutTs=1681606527556 2023-04-16 00:55:29,766 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting 157ff7be7222483aa909be17f0f6304b, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=374, earliestPutTs=1681606528765 2023-04-16 00:55:29,766 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting ab710e19efc841b298189bda8852f1ee, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=387, earliestPutTs=1681606528844 2023-04-16 00:55:29,773 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] throttle.PressureAwareThroughputController(145): ee2f682ad6dd0fe8e91c021d54d3bdc5#C#compaction#366 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-16 00:55:29,801 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230416405eb547e55c4c88afa4f0414c569e03 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230416405eb547e55c4c88afa4f0414c569e03 2023-04-16 00:55:29,885 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:29,885 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59666 deadline: 1681606589881, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:29,885 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:29,885 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 164 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59642 deadline: 1681606589882, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:29,891 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:29,891 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59650 deadline: 1681606589888, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:29,897 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:29,898 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59664 deadline: 1681606589892, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:30,189 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/.tmp/C/6bfa232107b046459a24b7697d4116ec as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/C/6bfa232107b046459a24b7697d4116ec 2023-04-16 00:55:30,191 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:30,191 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 159 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59666 deadline: 1681606590186, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:30,191 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:30,192 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 166 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59642 deadline: 1681606590186, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:30,200 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in ee2f682ad6dd0fe8e91c021d54d3bdc5/C of ee2f682ad6dd0fe8e91c021d54d3bdc5 into 6bfa232107b046459a24b7697d4116ec(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 00:55:30,201 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for ee2f682ad6dd0fe8e91c021d54d3bdc5: 2023-04-16 00:55:30,201 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5., storeName=ee2f682ad6dd0fe8e91c021d54d3bdc5/C, priority=13, startTime=1681606529329; duration=0sec 2023-04-16 00:55:30,201 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 00:55:30,201 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/.tmp/A/e9f7d4d51d1d45978e62a9839dba11c3 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/A/e9f7d4d51d1d45978e62a9839dba11c3 2023-04-16 00:55:30,202 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:30,203 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59650 deadline: 1681606590201, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:30,203 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:30,203 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59664 deadline: 1681606590201, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:30,206 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in ee2f682ad6dd0fe8e91c021d54d3bdc5/A of ee2f682ad6dd0fe8e91c021d54d3bdc5 into e9f7d4d51d1d45978e62a9839dba11c3(size=26.0 K), total size for store is 26.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 00:55:30,207 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for ee2f682ad6dd0fe8e91c021d54d3bdc5: 2023-04-16 00:55:30,207 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5., storeName=ee2f682ad6dd0fe8e91c021d54d3bdc5/A, priority=12, startTime=1681606529329; duration=0sec 2023-04-16 00:55:30,207 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 00:55:30,208 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=409, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/.tmp/A/47203eb368ac4ee58710963f6e345192 2023-04-16 00:55:30,625 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=409 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/.tmp/B/2883ee63702c42a2bc36dc63953f2384 2023-04-16 00:55:30,697 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:30,698 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 161 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59666 deadline: 1681606590695, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:30,699 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:30,699 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 168 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59642 deadline: 1681606590696, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:30,709 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:30,709 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59664 deadline: 1681606590707, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:30,709 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:30,709 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59650 deadline: 1681606590707, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:31,035 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=409 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/.tmp/C/ee7334928c3641a68884204f77c716e2 2023-04-16 00:55:31,039 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/.tmp/A/47203eb368ac4ee58710963f6e345192 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/A/47203eb368ac4ee58710963f6e345192 2023-04-16 00:55:31,043 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/A/47203eb368ac4ee58710963f6e345192, entries=150, sequenceid=409, filesize=25.4 K 2023-04-16 00:55:31,044 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/.tmp/B/2883ee63702c42a2bc36dc63953f2384 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/B/2883ee63702c42a2bc36dc63953f2384 2023-04-16 00:55:31,054 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/B/2883ee63702c42a2bc36dc63953f2384, entries=150, sequenceid=409, filesize=11.9 K 2023-04-16 00:55:31,055 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/.tmp/C/ee7334928c3641a68884204f77c716e2 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/C/ee7334928c3641a68884204f77c716e2 2023-04-16 00:55:31,058 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/C/ee7334928c3641a68884204f77c716e2, entries=150, sequenceid=409, filesize=11.9 K 2023-04-16 00:55:31,059 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for ee2f682ad6dd0fe8e91c021d54d3bdc5 in 1683ms, sequenceid=409, compaction requested=false 2023-04-16 00:55:31,059 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for ee2f682ad6dd0fe8e91c021d54d3bdc5: 2023-04-16 00:55:31,059 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. 2023-04-16 00:55:31,059 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:55:31,059 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:55:31,059 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:31,059 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:55:31,059 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:55:31,061 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:31,061 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:31,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:31,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:31,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:31,061 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:31,061 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:55:31,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:31,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:31,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:31,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:31,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:31,062 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:55:31,062 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:55:31,062 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@565347d4[Count = 0] remaining members to acquire global barrier 2023-04-16 00:55:31,062 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:31,063 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:31,063 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:31,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:31,064 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:55:31,064 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:55:31,064 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:55:31,068 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:55:31,068 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:31,068 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:55:31,068 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:31,069 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:55:31,069 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:55:31,069 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:55:31,069 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:55:31,069 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:55:31,069 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:55:31,069 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:55:31,070 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:31,070 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:31,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:31,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:31,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:31,070 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:31,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:31,070 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:31,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:31,071 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:31,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:31,071 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:31,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:31,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:31,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:31,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:31,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:31,072 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:31,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:31,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:31,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:31,074 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:31,074 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:31,074 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:31,074 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:31,075 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:31,075 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:31,075 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:31,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:31,075 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:55:31,075 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:55:31,075 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:31,075 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:31,075 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:31,075 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:31,075 DEBUG [Thread-2414] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:55:31,075 DEBUG [Thread-2414] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:55:31,075 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:31,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:31,175 DEBUG [Thread-2414] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:55:31,176 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:55:31,177 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:55:31,179 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:55:31,179 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:55:31,180 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:55:31,180 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:31,180 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:55:31,180 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:55:31,180 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:31,180 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:31,181 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:31,181 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:31,181 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:31,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:31,182 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:31,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:31,182 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:55:31,183 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:31,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:55:31,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:31,184 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:55:31,184 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:55:31,184 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:55:31,184 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:55:31,184 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:31,184 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:55:31,184 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:55:31,184 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:55:31,184 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. 2023-04-16 00:55:31,184 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. started... 2023-04-16 00:55:31,185 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing ee2f682ad6dd0fe8e91c021d54d3bdc5 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-16 00:55:31,185 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ee2f682ad6dd0fe8e91c021d54d3bdc5, store=A 2023-04-16 00:55:31,185 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:55:31,185 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ee2f682ad6dd0fe8e91c021d54d3bdc5, store=B 2023-04-16 00:55:31,185 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:55:31,185 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ee2f682ad6dd0fe8e91c021d54d3bdc5, store=C 2023-04-16 00:55:31,185 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:55:31,212 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304163b31b2f0f7b246f9837c73691e3c3c00 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304163b31b2f0f7b246f9837c73691e3c3c00 2023-04-16 00:55:31,225 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=424, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/.tmp/A/746a2ef59cfc4e9bac430d63a8efb03f 2023-04-16 00:55:31,642 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=424 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/.tmp/B/7b8ed50205274804ba5f8e96c9da8fa2 2023-04-16 00:55:31,706 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(9158): Flush requested on ee2f682ad6dd0fe8e91c021d54d3bdc5 2023-04-16 00:55:31,706 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. as already flushing 2023-04-16 00:55:31,740 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:31,740 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 172 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59664 deadline: 1681606591735, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:31,741 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681606531733, max timestamp=1681606531739Num uniques 859; 2023-04-16 00:55:31,741 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-16 00:55:31,741 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681606531733, max timestamp=1681606531739Num uniques 859; 2023-04-16 00:55:31,741 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-16 00:55:31,741 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681606531733, max timestamp=1681606531739Num uniques 859; 2023-04-16 00:55:31,741 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-16 00:55:31,774 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:31,774 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 183 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59650 deadline: 1681606591773, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:31,780 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:31,781 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 179 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59642 deadline: 1681606591774, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:31,781 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:31,781 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 174 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59666 deadline: 1681606591775, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:31,845 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:31,845 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 174 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59664 deadline: 1681606591841, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:31,878 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:31,878 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 185 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59650 deadline: 1681606591875, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:31,886 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:31,886 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 181 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59642 deadline: 1681606591881, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:31,886 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:31,886 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 176 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59666 deadline: 1681606591882, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:32,049 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:32,050 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 176 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59664 deadline: 1681606592046, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:32,051 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=424 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/.tmp/C/8b02f65560944dc5884bc589421eec3a 2023-04-16 00:55:32,063 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/.tmp/A/746a2ef59cfc4e9bac430d63a8efb03f as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/A/746a2ef59cfc4e9bac430d63a8efb03f 2023-04-16 00:55:32,067 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/A/746a2ef59cfc4e9bac430d63a8efb03f, entries=150, sequenceid=424, filesize=25.4 K 2023-04-16 00:55:32,067 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/.tmp/B/7b8ed50205274804ba5f8e96c9da8fa2 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/B/7b8ed50205274804ba5f8e96c9da8fa2 2023-04-16 00:55:32,071 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/B/7b8ed50205274804ba5f8e96c9da8fa2, entries=150, sequenceid=424, filesize=11.9 K 2023-04-16 00:55:32,072 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/.tmp/C/8b02f65560944dc5884bc589421eec3a as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/C/8b02f65560944dc5884bc589421eec3a 2023-04-16 00:55:32,076 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/C/8b02f65560944dc5884bc589421eec3a, entries=150, sequenceid=424, filesize=11.9 K 2023-04-16 00:55:32,077 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=120.76 KB/123660 for ee2f682ad6dd0fe8e91c021d54d3bdc5 in 892ms, sequenceid=424, compaction requested=true 2023-04-16 00:55:32,077 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for ee2f682ad6dd0fe8e91c021d54d3bdc5: 2023-04-16 00:55:32,077 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. 2023-04-16 00:55:32,077 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:55:32,077 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:55:32,077 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:32,078 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:55:32,078 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:55:32,080 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:32,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:32,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:32,080 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:32,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:32,080 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:32,081 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:55:32,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:32,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:32,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:32,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:32,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:32,082 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:55:32,082 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5bbc8221[Count = 0] remaining members to acquire global barrier 2023-04-16 00:55:32,082 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:55:32,082 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:32,083 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:32,083 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:32,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:32,083 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:55:32,083 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:55:32,083 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:55:32,083 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:32,083 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:55:32,084 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(9158): Flush requested on ee2f682ad6dd0fe8e91c021d54d3bdc5 2023-04-16 00:55:32,085 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing ee2f682ad6dd0fe8e91c021d54d3bdc5 3/3 column families, dataSize=127.47 KB heapSize=360.38 KB 2023-04-16 00:55:32,085 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ee2f682ad6dd0fe8e91c021d54d3bdc5, store=A 2023-04-16 00:55:32,085 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 00:55:32,085 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ee2f682ad6dd0fe8e91c021d54d3bdc5, store=B 2023-04-16 00:55:32,085 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 00:55:32,085 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ee2f682ad6dd0fe8e91c021d54d3bdc5, store=C 2023-04-16 00:55:32,085 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:32,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:32,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:32,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:32,085 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 00:55:32,085 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:55:32,085 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:32,085 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:55:32,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:32,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:32,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:32,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:32,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:32,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:32,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:32,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:55:32,088 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:55:32,088 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:55:32,088 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:55:32,088 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:55:32,088 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:55:32,088 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:55:32,091 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:32,091 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:32,091 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:32,091 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:32,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:32,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:32,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:32,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:32,092 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:32,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:32,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:32,092 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:32,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:32,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:32,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:32,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:32,093 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:32,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:32,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:32,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:32,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:32,096 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:32,096 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:32,096 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:32,096 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:55:32,096 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:32,096 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:32,096 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:32,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:32,096 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:32,096 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:32,096 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:32,096 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:32,096 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:32,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:32,098 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:32,098 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:55:32,098 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:55:32,098 DEBUG [Thread-2414] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:55:32,098 DEBUG [Thread-2414] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:55:32,106 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023041671c540bf6462428aaefbfb07db9cab0d to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041671c540bf6462428aaefbfb07db9cab0d 2023-04-16 00:55:32,113 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:32,113 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 180 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59666 deadline: 1681606592105, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:32,120 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:32,120 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 191 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59650 deadline: 1681606592113, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:32,126 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:32,126 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 186 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59642 deadline: 1681606592114, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:32,198 DEBUG [Thread-2414] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:55:32,199 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:55:32,200 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:55:32,201 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:55:32,201 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:55:32,201 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:55:32,201 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:32,201 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:55:32,201 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:55:32,202 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:32,202 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:32,203 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:32,203 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:32,203 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:32,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:32,203 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:32,203 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:55:32,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:32,204 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:32,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:55:32,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:32,204 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:55:32,204 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:55:32,204 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:55:32,204 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:55:32,204 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:32,204 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:55:32,205 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:55:32,205 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:55:32,205 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. 2023-04-16 00:55:32,205 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. started... 2023-04-16 00:55:32,205 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. as already flushing 2023-04-16 00:55:32,205 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. 2023-04-16 00:55:32,205 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:55:32,205 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:55:32,205 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:32,205 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:55:32,205 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:55:32,206 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:32,206 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:32,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:32,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:32,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:32,207 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:32,207 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:55:32,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:32,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:32,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:32,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:32,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:32,208 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:55:32,208 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@12a6a735[Count = 0] remaining members to acquire global barrier 2023-04-16 00:55:32,208 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:55:32,208 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:32,209 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:32,209 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:32,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:32,209 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:55:32,209 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:55:32,209 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:55:32,210 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:32,210 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:55:32,212 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:55:32,212 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:32,212 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:32,213 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:55:32,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:32,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:32,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:32,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:32,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:32,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:32,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:32,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:32,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:32,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:32,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:55:32,215 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:55:32,215 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:55:32,215 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:55:32,215 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:55:32,215 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:55:32,215 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:55:32,216 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:32,216 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:32,216 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:32,216 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:32,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:32,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:32,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:32,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:32,217 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:32,217 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:32,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:32,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:32,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:32,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:32,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:32,218 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:32,218 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:32,218 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 182 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59666 deadline: 1681606592214, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:32,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:32,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:32,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:32,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:32,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:32,220 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:32,220 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:32,220 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:32,220 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:32,220 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:32,220 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:32,220 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:32,220 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:32,220 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:55:32,220 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:55:32,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:32,220 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:32,220 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:32,220 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:32,220 DEBUG [Thread-2414] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:55:32,221 DEBUG [Thread-2414] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:55:32,221 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:55:32,221 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:32,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:32,225 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:32,225 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 193 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59650 deadline: 1681606592221, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:32,231 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:32,231 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 188 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59642 deadline: 1681606592227, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:32,321 DEBUG [Thread-2414] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:55:32,321 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:55:32,322 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:55:32,323 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:55:32,324 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:55:32,324 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:55:32,324 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:32,324 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:55:32,324 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:55:32,324 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:32,324 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:32,325 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:32,326 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:32,326 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:32,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:32,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:32,326 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:32,326 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:32,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:55:32,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:32,326 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:55:32,327 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:55:32,327 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:55:32,327 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:55:32,327 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:55:32,327 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:32,327 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:55:32,327 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:55:32,327 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:55:32,327 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. 2023-04-16 00:55:32,327 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. started... 2023-04-16 00:55:32,327 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. as already flushing 2023-04-16 00:55:32,327 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. 2023-04-16 00:55:32,327 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:55:32,327 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:55:32,327 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:32,327 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:55:32,327 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:55:32,334 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:32,334 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:32,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:32,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:32,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:32,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:32,334 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:32,334 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:55:32,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:32,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:32,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:32,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:32,336 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:55:32,336 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@53abc758[Count = 0] remaining members to acquire global barrier 2023-04-16 00:55:32,336 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:55:32,336 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:32,337 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:32,337 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:32,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:32,337 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:55:32,337 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:55:32,337 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:55:32,337 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:32,337 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:55:32,338 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:32,338 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:55:32,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:32,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:32,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:32,338 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:32,338 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:55:32,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:32,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:32,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:32,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:32,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:32,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:32,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:32,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:55:32,340 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:55:32,340 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:55:32,340 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:55:32,340 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:55:32,340 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:55:32,340 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:55:32,346 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:32,346 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:32,346 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:32,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:32,346 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:32,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:32,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:32,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:32,346 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:32,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:32,346 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:32,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:32,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:32,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:32,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:32,348 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:32,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:32,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:32,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:32,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:32,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:32,350 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:32,350 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:32,350 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:32,350 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:32,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:32,350 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:32,350 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:32,350 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:32,350 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:32,350 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:55:32,350 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:32,350 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:55:32,350 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:32,351 DEBUG [Thread-2414] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:55:32,351 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:32,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:32,350 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:32,351 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:55:32,351 DEBUG [Thread-2414] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:55:32,360 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:32,361 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 178 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59664 deadline: 1681606592353, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:32,423 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:32,423 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 184 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59666 deadline: 1681606592419, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:32,430 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:32,430 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 195 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59650 deadline: 1681606592427, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:32,437 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:32,437 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 190 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59642 deadline: 1681606592433, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:32,451 DEBUG [Thread-2414] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:55:32,452 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:55:32,453 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:55:32,454 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:55:32,454 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:55:32,455 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:55:32,455 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:32,455 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:55:32,455 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:55:32,455 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:32,455 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:32,456 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:32,456 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:32,456 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:32,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:32,457 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:32,457 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:55:32,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:32,458 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:32,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:55:32,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:32,459 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:55:32,459 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:55:32,459 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:55:32,459 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:55:32,459 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:32,459 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:55:32,460 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:55:32,460 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:55:32,460 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. 2023-04-16 00:55:32,461 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. started... 2023-04-16 00:55:32,461 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. as already flushing 2023-04-16 00:55:32,461 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. 2023-04-16 00:55:32,461 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:55:32,462 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:55:32,462 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:32,462 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:55:32,462 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:55:32,463 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:32,463 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:32,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:32,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:32,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:32,463 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:32,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:32,464 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:55:32,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:32,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:32,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:32,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:32,465 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:55:32,465 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1f0a571e[Count = 0] remaining members to acquire global barrier 2023-04-16 00:55:32,465 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:55:32,465 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:32,467 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:32,467 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:32,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:32,467 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:55:32,467 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:55:32,467 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:55:32,467 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:32,467 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:55:32,469 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:32,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:32,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:32,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:32,469 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:55:32,469 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:32,469 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:55:32,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:32,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:32,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:32,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:32,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:32,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:32,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:32,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:55:32,472 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:55:32,472 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:55:32,472 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:55:32,472 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:55:32,472 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:55:32,472 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:55:32,473 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:32,473 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:32,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:32,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:32,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:32,473 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:32,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:32,473 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:32,474 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:32,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:32,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:32,474 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:32,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:32,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:32,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:32,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:32,476 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:32,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:32,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:32,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:32,479 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:32,479 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:32,479 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:55:32,479 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:55:32,479 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:32,479 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:32,479 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:32,479 DEBUG [Thread-2414] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:55:32,479 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:32,479 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:32,480 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:32,480 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:32,479 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:32,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:32,479 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:32,480 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:55:32,479 DEBUG [Thread-2414] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:55:32,480 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:32,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:32,514 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=459, memsize=42.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/.tmp/A/7232901846364a55b5cf6b7877b7c105 2023-04-16 00:55:32,580 DEBUG [Thread-2414] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:55:32,580 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:55:32,582 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:55:32,583 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:55:32,583 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:55:32,583 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:55:32,583 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:32,583 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:55:32,583 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:55:32,584 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:32,584 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:32,586 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:32,586 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:32,586 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:32,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:32,586 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:32,586 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:55:32,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:32,587 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:32,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:55:32,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:32,587 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:55:32,587 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:55:32,587 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:55:32,587 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:55:32,587 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:32,587 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:55:32,587 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:55:32,587 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:55:32,587 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. 2023-04-16 00:55:32,588 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. started... 2023-04-16 00:55:32,588 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. as already flushing 2023-04-16 00:55:32,588 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. 2023-04-16 00:55:32,588 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:55:32,588 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:55:32,588 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:32,588 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:55:32,588 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:55:32,589 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:32,589 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:32,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:32,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:32,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:32,590 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:32,590 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:55:32,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:32,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:32,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:32,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:32,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:32,591 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:55:32,591 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@20187354[Count = 0] remaining members to acquire global barrier 2023-04-16 00:55:32,591 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:55:32,591 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:32,592 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:32,592 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:32,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:32,592 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:32,592 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:55:32,592 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:55:32,592 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:55:32,592 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:55:32,593 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:32,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:32,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:32,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:32,594 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:55:32,594 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:32,594 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:55:32,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:32,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:32,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:32,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:32,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:32,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:32,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:32,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:55:32,596 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:55:32,596 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:55:32,596 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:55:32,596 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:55:32,596 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:55:32,596 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:55:32,598 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:32,598 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:32,598 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:32,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:32,598 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:32,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:32,599 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:32,599 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:32,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:32,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:32,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:32,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:32,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:32,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:32,600 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:32,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:32,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:32,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:32,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:32,602 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:32,602 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:32,602 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:32,602 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:32,602 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:32,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:32,602 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:32,602 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:32,602 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:32,602 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:32,602 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:32,602 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:32,602 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:55:32,602 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:55:32,602 DEBUG [Thread-2414] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:55:32,602 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:55:32,602 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:32,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:32,602 DEBUG [Thread-2414] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:55:32,703 DEBUG [Thread-2414] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:55:32,703 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:55:32,705 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:55:32,706 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:55:32,706 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:55:32,706 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:55:32,706 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:32,706 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:55:32,706 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:55:32,706 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:32,706 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:32,708 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:32,708 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:32,708 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:32,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:32,708 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:32,708 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:55:32,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:32,708 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:32,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:55:32,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:32,709 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:55:32,709 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:55:32,709 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:55:32,709 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:55:32,709 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:32,709 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:55:32,709 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:55:32,709 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:55:32,709 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. 2023-04-16 00:55:32,709 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. started... 2023-04-16 00:55:32,709 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. as already flushing 2023-04-16 00:55:32,710 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. 2023-04-16 00:55:32,710 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:55:32,710 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:55:32,710 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:32,710 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:55:32,710 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:55:32,711 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:32,711 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:32,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:32,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:32,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:32,712 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:32,712 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:55:32,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:32,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:32,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:32,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:32,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:32,713 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:55:32,713 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7a0f76d[Count = 0] remaining members to acquire global barrier 2023-04-16 00:55:32,713 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:55:32,713 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:32,715 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:32,715 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:32,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:32,715 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:55:32,715 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:55:32,715 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:55:32,715 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:32,715 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:55:32,717 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:32,717 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:55:32,717 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:32,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:32,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:32,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:32,717 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:55:32,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:32,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:32,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:32,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:32,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:32,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:32,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:32,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:55:32,719 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:55:32,719 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:55:32,719 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:55:32,719 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:55:32,719 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:55:32,719 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:55:32,720 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:32,720 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:32,720 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:32,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:32,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:32,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:32,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:32,720 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:32,721 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:32,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:32,721 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:32,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:32,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:32,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:32,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:32,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:32,722 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:32,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:32,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:32,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:32,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:32,730 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:32,730 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:32,730 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:32,730 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:32,730 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:55:32,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:32,730 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:32,730 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:32,730 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:32,731 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:32,731 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:32,730 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:32,730 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:32,731 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:55:32,730 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:55:32,731 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:32,731 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:32,731 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 186 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59666 deadline: 1681606592725, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:32,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:32,731 DEBUG [Thread-2414] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:55:32,731 DEBUG [Thread-2414] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:55:32,733 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:32,734 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 197 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59650 deadline: 1681606592731, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:32,740 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:32,741 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 192 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59642 deadline: 1681606592738, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:32,831 DEBUG [Thread-2414] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:55:32,832 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:55:32,833 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:55:32,834 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:55:32,834 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:55:32,834 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:55:32,834 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:32,834 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:55:32,834 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:55:32,835 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:32,835 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:32,836 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:32,836 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:32,836 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:32,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:32,837 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:32,837 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:55:32,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:32,837 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:32,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:55:32,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:32,837 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:55:32,838 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:55:32,838 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:55:32,838 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:55:32,838 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:32,838 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:55:32,838 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:55:32,838 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:55:32,838 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. 2023-04-16 00:55:32,838 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. started... 2023-04-16 00:55:32,838 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. as already flushing 2023-04-16 00:55:32,838 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. 2023-04-16 00:55:32,838 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:55:32,838 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:55:32,838 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:32,838 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:55:32,838 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:55:32,839 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:32,839 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:32,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:32,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:32,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:32,840 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:32,840 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:55:32,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:32,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:32,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:32,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:32,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:32,841 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:55:32,841 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@abc71c6[Count = 0] remaining members to acquire global barrier 2023-04-16 00:55:32,841 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:55:32,841 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:32,842 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:32,842 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:32,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:32,842 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:55:32,842 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:55:32,842 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:55:32,843 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:32,843 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:55:32,845 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:55:32,845 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:32,845 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:32,845 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:55:32,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:32,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:32,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:32,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:32,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:32,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:32,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:32,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:32,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:32,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:32,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:55:32,847 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:55:32,847 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:55:32,847 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:55:32,847 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:55:32,847 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:55:32,847 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:55:32,849 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:32,849 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:32,849 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:32,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:32,849 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:32,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:32,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:32,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:32,849 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:32,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:32,849 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:32,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:32,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:32,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:32,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:32,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:32,851 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:32,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:32,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:32,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:32,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:32,853 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:32,853 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:32,853 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:32,853 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:32,853 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:32,853 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:32,853 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:32,853 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:32,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:32,853 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:32,853 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:55:32,854 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:55:32,854 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:32,854 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:32,854 DEBUG [Thread-2414] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:55:32,854 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:32,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:32,854 DEBUG [Thread-2414] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:55:32,854 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:55:32,868 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:32,868 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 180 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59664 deadline: 1681606592865, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:32,929 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=459 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/.tmp/B/1858cee89d46435da7e61e248a57e0ac 2023-04-16 00:55:32,954 DEBUG [Thread-2414] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:55:32,955 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:55:32,956 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:55:32,957 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:55:32,957 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:55:32,957 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:55:32,957 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:32,957 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:55:32,957 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:55:32,957 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:32,957 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:32,958 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:32,958 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:32,958 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:32,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:32,959 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:32,959 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:55:32,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:32,959 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:32,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:55:32,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:32,960 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:55:32,960 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:55:32,960 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:55:32,960 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:55:32,960 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:32,960 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:55:32,960 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:55:32,960 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:55:32,960 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. 2023-04-16 00:55:32,960 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. started... 2023-04-16 00:55:32,960 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. as already flushing 2023-04-16 00:55:32,960 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. 2023-04-16 00:55:32,960 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:55:32,960 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:55:32,960 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:32,960 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:55:32,961 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:55:32,962 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:32,962 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:32,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:32,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:32,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:32,963 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:32,963 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:55:32,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:32,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:32,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:32,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:32,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:32,964 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:55:32,964 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@47436528[Count = 0] remaining members to acquire global barrier 2023-04-16 00:55:32,964 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:55:32,964 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:32,966 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:32,966 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:55:32,966 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:32,966 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:32,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:32,967 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:55:32,967 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:55:32,967 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:55:32,968 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:55:32,968 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:32,968 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:32,968 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:55:32,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:32,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:32,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:32,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:32,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:32,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:32,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:32,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:32,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:32,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:32,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:55:32,970 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:55:32,970 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:55:32,970 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:55:32,970 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:55:32,970 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:55:32,970 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:55:32,972 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:32,972 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:32,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:32,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:32,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:32,972 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:32,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:32,972 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:32,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:32,972 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:32,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:32,973 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:32,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:32,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:32,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:32,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:32,974 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:32,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:32,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:32,977 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:32,977 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:32,977 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:32,977 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:32,977 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:32,977 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:32,977 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:32,977 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:32,977 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:55:32,977 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:55:32,977 DEBUG [Thread-2414] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:55:32,977 DEBUG [Thread-2414] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:55:32,979 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:55:32,979 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:32,979 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:32,979 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:32,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:32,980 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:32,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:33,078 DEBUG [Thread-2414] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:55:33,078 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:55:33,080 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:55:33,081 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:55:33,081 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:55:33,081 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:55:33,081 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:33,081 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:55:33,081 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:55:33,082 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:33,082 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:33,083 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:33,083 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:33,083 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:33,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:33,083 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:33,083 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:55:33,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:33,084 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:33,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:55:33,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:33,084 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:55:33,084 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:55:33,084 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:55:33,084 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:55:33,084 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:33,084 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:55:33,085 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:55:33,085 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:55:33,085 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. 2023-04-16 00:55:33,085 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. started... 2023-04-16 00:55:33,085 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. as already flushing 2023-04-16 00:55:33,085 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. 2023-04-16 00:55:33,085 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:55:33,085 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:55:33,085 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:33,085 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:55:33,085 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:55:33,086 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:33,086 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:33,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:33,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:33,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:33,087 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:33,087 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:55:33,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:33,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:33,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:33,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:33,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:33,088 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:55:33,088 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@24a23c47[Count = 0] remaining members to acquire global barrier 2023-04-16 00:55:33,088 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:55:33,088 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:33,089 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:33,089 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:33,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:33,089 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:55:33,089 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:55:33,089 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:55:33,089 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:33,089 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:55:33,092 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:33,092 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:55:33,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:33,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:33,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:33,092 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:33,092 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:55:33,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:33,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:33,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:33,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:33,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:33,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:33,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:33,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:55:33,094 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:55:33,094 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:55:33,094 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:55:33,094 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:55:33,094 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:55:33,094 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:55:33,096 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:33,096 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:33,096 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:33,096 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:33,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:33,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:33,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:33,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:33,096 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:33,096 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:33,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:33,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:33,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:33,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:33,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:33,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:33,097 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:33,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:33,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:33,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:33,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:33,102 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:33,103 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:33,103 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:33,103 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:55:33,103 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:33,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:33,103 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:33,103 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:55:33,103 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:55:33,103 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:33,103 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:33,103 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:33,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:33,103 DEBUG [Thread-2414] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:55:33,103 DEBUG [Thread-2414] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:55:33,103 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:33,103 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:33,103 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:33,103 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:33,203 DEBUG [Thread-2414] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:55:33,204 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:55:33,205 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:55:33,206 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:55:33,206 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:55:33,206 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:55:33,206 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:33,206 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:55:33,206 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:55:33,207 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:33,207 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:33,209 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:33,209 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:33,209 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:33,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:33,209 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:33,209 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:55:33,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:33,209 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:33,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:55:33,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:33,210 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:55:33,210 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:55:33,210 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:55:33,210 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:55:33,210 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:33,210 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:55:33,210 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:55:33,210 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:55:33,210 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. 2023-04-16 00:55:33,210 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. started... 2023-04-16 00:55:33,211 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. as already flushing 2023-04-16 00:55:33,211 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. 2023-04-16 00:55:33,211 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:55:33,211 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:55:33,211 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:33,211 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:55:33,211 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:55:33,212 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:33,212 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:33,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:33,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:33,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:33,212 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:33,212 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:55:33,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:33,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:33,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:33,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:33,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:33,214 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:55:33,214 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2439bc21[Count = 0] remaining members to acquire global barrier 2023-04-16 00:55:33,214 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:55:33,214 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:33,215 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:33,215 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:33,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:33,215 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:33,215 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:55:33,215 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:55:33,215 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:55:33,215 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:55:33,216 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:55:33,216 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:33,216 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:33,216 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:55:33,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:33,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:33,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:33,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:33,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:33,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:33,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:33,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:33,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:33,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:33,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:55:33,218 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:55:33,218 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:55:33,218 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:55:33,218 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:55:33,218 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:55:33,218 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:55:33,221 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:33,221 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:33,221 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:33,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:33,221 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:33,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:33,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:33,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:33,221 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:33,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:33,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:33,222 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:33,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:33,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:33,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:33,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:33,223 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:33,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:33,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:33,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:33,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:33,224 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:33,224 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:33,224 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:33,224 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:33,224 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:33,224 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:33,224 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:55:33,225 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:55:33,225 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:55:33,224 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:33,224 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:33,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:33,225 DEBUG [Thread-2414] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:55:33,225 DEBUG [Thread-2414] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:55:33,225 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:33,225 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:33,225 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:33,225 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:33,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:33,238 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:33,238 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 188 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59666 deadline: 1681606593234, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:33,242 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:33,242 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 199 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59650 deadline: 1681606593237, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:33,245 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:33,245 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 194 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59642 deadline: 1681606593243, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:33,325 DEBUG [Thread-2414] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:55:33,325 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:55:33,327 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:55:33,327 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:55:33,328 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:55:33,328 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:55:33,328 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:33,328 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:55:33,328 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:55:33,328 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:33,328 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:33,329 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:33,329 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:33,329 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:33,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:33,330 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:33,330 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:55:33,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:33,330 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:33,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:55:33,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:33,330 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:55:33,330 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:55:33,331 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:55:33,331 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:55:33,331 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:33,331 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:55:33,331 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:55:33,331 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:55:33,331 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. 2023-04-16 00:55:33,331 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. started... 2023-04-16 00:55:33,331 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. as already flushing 2023-04-16 00:55:33,331 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. 2023-04-16 00:55:33,331 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:55:33,331 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:55:33,331 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:33,331 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:55:33,331 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:55:33,332 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:33,332 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:33,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:33,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:33,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:33,333 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:33,333 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:55:33,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:33,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:33,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:33,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:33,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:33,335 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:55:33,335 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4de2db21[Count = 0] remaining members to acquire global barrier 2023-04-16 00:55:33,335 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:55:33,335 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:33,338 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:33,338 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:33,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:33,338 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:55:33,338 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:55:33,338 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:33,338 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:55:33,338 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:55:33,339 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=459 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/.tmp/C/ff4d879155a94de0a34ace3c7b7def23 2023-04-16 00:55:33,339 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:33,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:33,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:33,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:33,339 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:55:33,340 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:33,340 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:55:33,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:33,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:33,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:33,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:33,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:33,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:33,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:33,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:55:33,342 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:55:33,342 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:55:33,343 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:55:33,343 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:55:33,343 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:55:33,343 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:55:33,343 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/.tmp/A/7232901846364a55b5cf6b7877b7c105 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/A/7232901846364a55b5cf6b7877b7c105 2023-04-16 00:55:33,344 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:33,344 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:33,344 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:33,344 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:33,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:33,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:33,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:33,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:33,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:33,346 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:33,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:33,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:33,346 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:33,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:33,347 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/A/7232901846364a55b5cf6b7877b7c105, entries=200, sequenceid=459, filesize=32.3 K 2023-04-16 00:55:33,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:33,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:33,347 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/.tmp/B/1858cee89d46435da7e61e248a57e0ac as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/B/1858cee89d46435da7e61e248a57e0ac 2023-04-16 00:55:33,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:33,348 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:33,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:33,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:33,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:33,350 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:33,350 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:33,350 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:33,350 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:33,350 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:33,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:33,350 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:33,350 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:55:33,351 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:55:33,350 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:33,351 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:33,351 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:33,351 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:33,351 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:33,351 DEBUG [Thread-2414] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:55:33,351 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:33,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:33,351 DEBUG [Thread-2414] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:55:33,351 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/B/1858cee89d46435da7e61e248a57e0ac, entries=150, sequenceid=459, filesize=11.9 K 2023-04-16 00:55:33,352 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/.tmp/C/ff4d879155a94de0a34ace3c7b7def23 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/C/ff4d879155a94de0a34ace3c7b7def23 2023-04-16 00:55:33,365 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/C/ff4d879155a94de0a34ace3c7b7def23, entries=150, sequenceid=459, filesize=11.9 K 2023-04-16 00:55:33,371 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~360.33 KB/368976, currentSize=53.67 KB/54960 for ee2f682ad6dd0fe8e91c021d54d3bdc5 in 1286ms, sequenceid=459, compaction requested=true 2023-04-16 00:55:33,371 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for ee2f682ad6dd0fe8e91c021d54d3bdc5: 2023-04-16 00:55:33,371 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 00:55:33,371 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 00:55:33,371 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-16 00:55:33,371 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-16 00:55:33,371 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 00:55:33,375 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49290 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-16 00:55:33,375 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 111770 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-16 00:55:33,375 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HStore(1912): ee2f682ad6dd0fe8e91c021d54d3bdc5/B is initiating minor compaction (all files) 2023-04-16 00:55:33,375 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HStore(1912): ee2f682ad6dd0fe8e91c021d54d3bdc5/A is initiating minor compaction (all files) 2023-04-16 00:55:33,375 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HRegion(2259): Starting compaction of ee2f682ad6dd0fe8e91c021d54d3bdc5/B in TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. 2023-04-16 00:55:33,375 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of ee2f682ad6dd0fe8e91c021d54d3bdc5/A in TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. 2023-04-16 00:55:33,375 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/B/529f111b0dd2430387ad55fd414e7016, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/B/2883ee63702c42a2bc36dc63953f2384, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/B/7b8ed50205274804ba5f8e96c9da8fa2, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/B/1858cee89d46435da7e61e248a57e0ac] into tmpdir=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/.tmp, totalSize=48.1 K 2023-04-16 00:55:33,375 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/A/e9f7d4d51d1d45978e62a9839dba11c3, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/A/47203eb368ac4ee58710963f6e345192, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/A/746a2ef59cfc4e9bac430d63a8efb03f, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/A/7232901846364a55b5cf6b7877b7c105] into tmpdir=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/.tmp, totalSize=109.2 K 2023-04-16 00:55:33,376 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting 529f111b0dd2430387ad55fd414e7016, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=387, earliestPutTs=1681606528844 2023-04-16 00:55:33,376 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.Compactor(207): Compacting e9f7d4d51d1d45978e62a9839dba11c3, keycount=150, bloomtype=ROW, size=26.0 K, encoding=NONE, compression=NONE, seqNum=387, earliestPutTs=1681606528844 2023-04-16 00:55:33,376 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting 2883ee63702c42a2bc36dc63953f2384, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=409, earliestPutTs=1681606528895 2023-04-16 00:55:33,376 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.Compactor(207): Compacting 47203eb368ac4ee58710963f6e345192, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=409, earliestPutTs=1681606528895 2023-04-16 00:55:33,377 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting 7b8ed50205274804ba5f8e96c9da8fa2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=424, earliestPutTs=1681606529565 2023-04-16 00:55:33,377 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.Compactor(207): Compacting 746a2ef59cfc4e9bac430d63a8efb03f, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=424, earliestPutTs=1681606529565 2023-04-16 00:55:33,377 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting 1858cee89d46435da7e61e248a57e0ac, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=459, earliestPutTs=1681606531766 2023-04-16 00:55:33,377 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.Compactor(207): Compacting 7232901846364a55b5cf6b7877b7c105, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=459, earliestPutTs=1681606531766 2023-04-16 00:55:33,409 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] throttle.PressureAwareThroughputController(145): ee2f682ad6dd0fe8e91c021d54d3bdc5#B#compaction#376 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-16 00:55:33,411 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] throttle.PressureAwareThroughputController(145): ee2f682ad6dd0fe8e91c021d54d3bdc5#A#compaction#375 average throughput is 1.23 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-16 00:55:33,424 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/.tmp/B/390d7fcb0fb14b6f84f6debe382d4c94 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/B/390d7fcb0fb14b6f84f6debe382d4c94 2023-04-16 00:55:33,428 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in ee2f682ad6dd0fe8e91c021d54d3bdc5/B of ee2f682ad6dd0fe8e91c021d54d3bdc5 into 390d7fcb0fb14b6f84f6debe382d4c94(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 00:55:33,428 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for ee2f682ad6dd0fe8e91c021d54d3bdc5: 2023-04-16 00:55:33,428 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5., storeName=ee2f682ad6dd0fe8e91c021d54d3bdc5/B, priority=12, startTime=1681606533371; duration=0sec 2023-04-16 00:55:33,428 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 00:55:33,429 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-16 00:55:33,430 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49290 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-16 00:55:33,430 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HStore(1912): ee2f682ad6dd0fe8e91c021d54d3bdc5/C is initiating minor compaction (all files) 2023-04-16 00:55:33,430 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HRegion(2259): Starting compaction of ee2f682ad6dd0fe8e91c021d54d3bdc5/C in TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. 2023-04-16 00:55:33,430 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/C/6bfa232107b046459a24b7697d4116ec, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/C/ee7334928c3641a68884204f77c716e2, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/C/8b02f65560944dc5884bc589421eec3a, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/C/ff4d879155a94de0a34ace3c7b7def23] into tmpdir=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/.tmp, totalSize=48.1 K 2023-04-16 00:55:33,430 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting 6bfa232107b046459a24b7697d4116ec, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=387, earliestPutTs=1681606528844 2023-04-16 00:55:33,430 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting ee7334928c3641a68884204f77c716e2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=409, earliestPutTs=1681606528895 2023-04-16 00:55:33,431 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting 8b02f65560944dc5884bc589421eec3a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=424, earliestPutTs=1681606529565 2023-04-16 00:55:33,431 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting ff4d879155a94de0a34ace3c7b7def23, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=459, earliestPutTs=1681606531766 2023-04-16 00:55:33,446 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] throttle.PressureAwareThroughputController(145): ee2f682ad6dd0fe8e91c021d54d3bdc5#C#compaction#377 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-16 00:55:33,451 DEBUG [Thread-2414] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:55:33,452 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:55:33,454 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:55:33,476 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:55:33,476 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:55:33,476 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:55:33,476 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:33,476 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:55:33,476 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:55:33,476 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:33,477 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:33,480 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:33,480 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:33,480 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:33,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:33,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:33,481 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:33,481 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:55:33,481 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:33,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:55:33,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:33,481 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:55:33,482 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:55:33,482 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:55:33,482 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:55:33,482 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:33,482 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:55:33,482 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:55:33,482 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:55:33,482 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. 2023-04-16 00:55:33,482 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. started... 2023-04-16 00:55:33,482 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing ee2f682ad6dd0fe8e91c021d54d3bdc5 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-16 00:55:33,482 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ee2f682ad6dd0fe8e91c021d54d3bdc5, store=A 2023-04-16 00:55:33,482 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:55:33,482 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ee2f682ad6dd0fe8e91c021d54d3bdc5, store=B 2023-04-16 00:55:33,483 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:55:33,483 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ee2f682ad6dd0fe8e91c021d54d3bdc5, store=C 2023-04-16 00:55:33,483 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:55:33,497 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304163e6d6685152f4c508d0b57ecaebca119 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304163e6d6685152f4c508d0b57ecaebca119 2023-04-16 00:55:33,870 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/.tmp/C/0288484dfc034b5eb668ba595be9916c as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/C/0288484dfc034b5eb668ba595be9916c 2023-04-16 00:55:33,874 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in ee2f682ad6dd0fe8e91c021d54d3bdc5/C of ee2f682ad6dd0fe8e91c021d54d3bdc5 into 0288484dfc034b5eb668ba595be9916c(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 00:55:33,874 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for ee2f682ad6dd0fe8e91c021d54d3bdc5: 2023-04-16 00:55:33,874 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5., storeName=ee2f682ad6dd0fe8e91c021d54d3bdc5/C, priority=12, startTime=1681606533371; duration=0sec 2023-04-16 00:55:33,874 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 00:55:33,890 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(9158): Flush requested on ee2f682ad6dd0fe8e91c021d54d3bdc5 2023-04-16 00:55:33,890 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. as already flushing 2023-04-16 00:55:33,905 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=471, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/.tmp/A/f0d3276b2571448fa3f2cb37aa8c9b1f 2023-04-16 00:55:33,924 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=471 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/.tmp/B/77f861403e0d4883be8ae5d76a678211 2023-04-16 00:55:34,008 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681606533949, max timestamp=1681606533994Num uniques 859; 2023-04-16 00:55:34,008 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681606533949, max timestamp=1681606533994Num uniques 859; 2023-04-16 00:55:34,008 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681606533949, max timestamp=1681606533994Num uniques 859; 2023-04-16 00:55:34,008 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-16 00:55:34,009 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-16 00:55:34,009 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-16 00:55:34,089 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:34,089 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 215 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59664 deadline: 1681606594089, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:34,195 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:34,195 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 217 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59664 deadline: 1681606594190, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:34,243 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:34,243 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 190 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59666 deadline: 1681606594239, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:34,249 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:34,249 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 201 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59650 deadline: 1681606594247, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:34,256 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:34,256 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 196 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59642 deadline: 1681606594253, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:34,256 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/.tmp/A/3db9e520b64e4dcb962f0c9eeda597f2 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/A/3db9e520b64e4dcb962f0c9eeda597f2 2023-04-16 00:55:34,260 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in ee2f682ad6dd0fe8e91c021d54d3bdc5/A of ee2f682ad6dd0fe8e91c021d54d3bdc5 into 3db9e520b64e4dcb962f0c9eeda597f2(size=26.1 K), total size for store is 26.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 00:55:34,260 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for ee2f682ad6dd0fe8e91c021d54d3bdc5: 2023-04-16 00:55:34,260 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5., storeName=ee2f682ad6dd0fe8e91c021d54d3bdc5/A, priority=12, startTime=1681606533371; duration=0sec 2023-04-16 00:55:34,260 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 00:55:34,336 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=471 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/.tmp/C/08fd47939e6746dda979eb423fc01c8f 2023-04-16 00:55:34,340 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/.tmp/A/f0d3276b2571448fa3f2cb37aa8c9b1f as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/A/f0d3276b2571448fa3f2cb37aa8c9b1f 2023-04-16 00:55:34,343 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/A/f0d3276b2571448fa3f2cb37aa8c9b1f, entries=100, sequenceid=471, filesize=18.6 K 2023-04-16 00:55:34,343 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/.tmp/B/77f861403e0d4883be8ae5d76a678211 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/B/77f861403e0d4883be8ae5d76a678211 2023-04-16 00:55:34,347 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/B/77f861403e0d4883be8ae5d76a678211, entries=100, sequenceid=471, filesize=9.5 K 2023-04-16 00:55:34,347 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/.tmp/C/08fd47939e6746dda979eb423fc01c8f as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/C/08fd47939e6746dda979eb423fc01c8f 2023-04-16 00:55:34,350 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/C/08fd47939e6746dda979eb423fc01c8f, entries=100, sequenceid=471, filesize=9.5 K 2023-04-16 00:55:34,351 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for ee2f682ad6dd0fe8e91c021d54d3bdc5 in 869ms, sequenceid=471, compaction requested=false 2023-04-16 00:55:34,351 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for ee2f682ad6dd0fe8e91c021d54d3bdc5: 2023-04-16 00:55:34,351 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. 2023-04-16 00:55:34,351 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:55:34,351 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:55:34,351 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:34,351 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:55:34,351 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:55:34,352 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:34,352 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:34,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:34,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:34,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:34,353 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:34,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:34,353 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:55:34,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:34,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:34,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:34,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:34,354 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:55:34,354 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@11d49d90[Count = 0] remaining members to acquire global barrier 2023-04-16 00:55:34,354 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:55:34,354 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:34,355 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:34,355 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:34,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:34,355 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:55:34,355 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:55:34,355 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:55:34,355 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:34,356 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:55:34,357 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:55:34,357 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:34,357 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:34,357 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:55:34,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:34,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:34,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:34,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:34,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:34,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:34,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:34,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:34,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:34,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:34,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:55:34,359 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:55:34,359 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:55:34,359 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:55:34,359 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:55:34,359 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:55:34,359 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:55:34,362 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:34,362 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:34,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:34,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:34,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:34,362 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:34,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:34,362 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:34,362 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:34,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:34,362 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:34,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:34,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:34,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:34,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:34,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:34,363 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:34,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:34,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:34,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:34,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:34,366 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:34,366 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:34,366 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:34,367 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:34,367 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:34,367 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:55:34,367 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:55:34,367 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:34,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:34,367 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:34,367 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:34,367 DEBUG [Thread-2414] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:55:34,367 DEBUG [Thread-2414] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:55:34,367 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:34,367 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:34,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:34,367 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:34,367 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:34,399 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(9158): Flush requested on ee2f682ad6dd0fe8e91c021d54d3bdc5 2023-04-16 00:55:34,399 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing ee2f682ad6dd0fe8e91c021d54d3bdc5 3/3 column families, dataSize=134.18 KB heapSize=384.98 KB 2023-04-16 00:55:34,399 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ee2f682ad6dd0fe8e91c021d54d3bdc5, store=A 2023-04-16 00:55:34,399 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 00:55:34,399 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ee2f682ad6dd0fe8e91c021d54d3bdc5, store=B 2023-04-16 00:55:34,399 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 00:55:34,399 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ee2f682ad6dd0fe8e91c021d54d3bdc5, store=C 2023-04-16 00:55:34,399 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 00:55:34,410 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023041682d926f025344c3cac852cd356a261e0 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041682d926f025344c3cac852cd356a261e0 2023-04-16 00:55:34,418 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=510, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/.tmp/A/dbcbd7228f01411289b79cc1e02db3c8 2023-04-16 00:55:34,453 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:34,453 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 227 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59664 deadline: 1681606594445, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:34,467 DEBUG [Thread-2414] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:55:34,467 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:55:34,469 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:55:34,469 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:55:34,469 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:55:34,470 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:55:34,470 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:34,470 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:55:34,470 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:55:34,470 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:34,470 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:34,471 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:34,471 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:34,471 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:34,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:34,471 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:34,471 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:55:34,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:34,472 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:34,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:55:34,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:34,472 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:55:34,472 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:55:34,472 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:55:34,473 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:55:34,473 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:34,473 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:55:34,473 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:55:34,473 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:55:34,473 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. 2023-04-16 00:55:34,473 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. started... 2023-04-16 00:55:34,473 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. as already flushing 2023-04-16 00:55:34,473 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. 2023-04-16 00:55:34,473 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:55:34,473 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:55:34,473 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:34,473 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:55:34,473 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:55:34,479 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:34,479 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:34,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:34,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:34,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:34,479 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:34,479 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:55:34,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:34,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:34,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:34,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:34,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:34,480 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:55:34,480 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@24a1dbe9[Count = 0] remaining members to acquire global barrier 2023-04-16 00:55:34,480 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:55:34,480 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:34,481 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:34,481 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:34,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:34,481 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:55:34,481 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:55:34,481 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:55:34,481 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:34,481 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:55:34,482 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:55:34,482 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:34,482 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:34,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:34,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:34,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:34,482 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:55:34,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:34,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:34,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:34,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:34,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:34,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:34,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:34,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:55:34,485 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:55:34,485 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:55:34,485 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:55:34,485 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:55:34,485 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:55:34,485 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:55:34,486 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:34,486 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:34,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:34,486 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:34,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:34,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:34,486 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:34,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:34,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:34,487 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:34,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:34,487 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:34,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:34,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:34,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:34,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:34,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:34,488 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:34,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:34,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:34,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:34,491 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:34,491 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:34,491 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:34,491 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:34,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:34,491 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:55:34,491 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:55:34,491 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:34,491 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:34,491 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:34,491 DEBUG [Thread-2414] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:55:34,491 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:34,491 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:34,491 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:34,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:34,491 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:34,491 DEBUG [Thread-2414] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:55:34,491 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:34,559 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:34,559 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 229 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59664 deadline: 1681606594554, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:34,591 DEBUG [Thread-2414] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:55:34,592 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:55:34,593 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:55:34,594 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:55:34,594 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:55:34,594 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:55:34,594 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:34,594 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:55:34,594 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:55:34,595 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:34,595 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:34,596 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:34,596 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:34,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:34,596 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:34,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:34,596 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:34,596 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:55:34,597 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:34,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:55:34,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:34,597 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:55:34,597 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:55:34,597 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:55:34,597 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:55:34,597 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:34,597 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:55:34,598 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:55:34,598 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:55:34,598 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. 2023-04-16 00:55:34,598 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. started... 2023-04-16 00:55:34,598 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. as already flushing 2023-04-16 00:55:34,598 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. 2023-04-16 00:55:34,598 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:55:34,598 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:55:34,598 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:34,598 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:55:34,598 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:55:34,599 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:34,599 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:34,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:34,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:34,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:34,600 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:34,600 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:55:34,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:34,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:34,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:34,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:34,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:34,601 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:55:34,601 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4d10b3e8[Count = 0] remaining members to acquire global barrier 2023-04-16 00:55:34,601 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:55:34,601 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:34,602 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:34,602 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:34,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:34,602 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:55:34,602 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:55:34,602 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:34,602 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:55:34,602 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:55:34,605 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:55:34,605 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:34,605 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:34,605 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:55:34,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:34,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:34,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:34,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:34,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:34,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:34,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:34,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:34,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:34,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:34,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:55:34,607 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:55:34,607 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:55:34,607 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:55:34,607 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:55:34,607 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:55:34,607 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:55:34,609 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:34,609 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:34,609 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:34,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:34,609 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:34,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:34,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:34,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:34,610 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:34,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:34,610 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:34,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:34,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:34,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:34,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:34,611 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:34,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:34,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:34,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:34,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:34,613 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:34,613 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:34,613 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:34,613 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:34,613 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:34,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:34,613 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:34,613 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:34,613 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:34,613 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:55:34,613 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:55:34,613 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:34,613 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:34,613 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:34,613 DEBUG [Thread-2414] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:55:34,613 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:34,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:34,613 DEBUG [Thread-2414] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:55:34,619 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:55:34,714 DEBUG [Thread-2414] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:55:34,714 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:55:34,715 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:55:34,716 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:55:34,717 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:55:34,717 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:55:34,717 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:34,717 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:55:34,717 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:55:34,717 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:34,717 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:34,718 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:34,718 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:34,718 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:34,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:34,719 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:34,719 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:55:34,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:34,719 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:34,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:55:34,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:34,719 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:55:34,720 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:55:34,720 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:55:34,720 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:55:34,720 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:34,720 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:55:34,720 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:55:34,720 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:55:34,720 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. 2023-04-16 00:55:34,720 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. started... 2023-04-16 00:55:34,720 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. as already flushing 2023-04-16 00:55:34,720 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. 2023-04-16 00:55:34,720 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:55:34,720 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:55:34,720 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:34,720 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:55:34,720 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:55:34,723 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:34,723 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:34,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:34,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:34,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:34,723 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:34,723 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:55:34,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:34,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:34,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:34,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:34,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:34,724 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:55:34,724 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@167052b8[Count = 0] remaining members to acquire global barrier 2023-04-16 00:55:34,724 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:55:34,724 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:34,725 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:34,725 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:34,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:34,726 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:55:34,726 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:55:34,726 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:34,726 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:55:34,726 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:55:34,727 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:34,727 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:55:34,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:34,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:34,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:34,727 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:34,727 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:55:34,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:34,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:34,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:34,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:34,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:34,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:34,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:34,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:55:34,729 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:55:34,729 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:55:34,729 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:55:34,729 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:55:34,729 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:55:34,729 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:55:34,730 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:34,730 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:34,730 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:34,730 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:34,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:34,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:34,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:34,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:34,731 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:34,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:34,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:34,731 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:34,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:34,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:34,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:34,732 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:34,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:34,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:34,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:34,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:34,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:34,735 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:34,735 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:34,735 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:34,735 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:34,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:34,735 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:34,735 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:34,735 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:34,735 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:34,735 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:34,735 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:34,735 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:55:34,735 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:55:34,735 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:34,736 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:55:34,735 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:34,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:34,736 DEBUG [Thread-2414] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:55:34,736 DEBUG [Thread-2414] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:55:34,765 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:34,766 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 231 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59664 deadline: 1681606594760, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:34,830 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=510 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/.tmp/B/2d0610a124d14e25a02244de2de67474 2023-04-16 00:55:34,836 DEBUG [Thread-2414] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:55:34,836 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:55:34,838 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:55:34,839 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:55:34,839 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:55:34,839 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:55:34,839 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:34,839 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:55:34,839 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:55:34,840 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:34,840 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:34,841 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:34,841 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:34,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:34,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:34,842 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:34,842 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:34,842 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:34,842 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:55:34,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:55:34,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:34,843 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:55:34,843 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:55:34,843 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:55:34,843 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:55:34,843 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:34,843 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:55:34,843 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:55:34,843 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. 2023-04-16 00:55:34,843 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. started... 2023-04-16 00:55:34,843 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:55:34,843 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. as already flushing 2023-04-16 00:55:34,843 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. 2023-04-16 00:55:34,843 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:55:34,843 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:55:34,843 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:34,843 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:55:34,843 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:55:34,845 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:34,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:34,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:34,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:34,845 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:34,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:34,846 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:34,846 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:55:34,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:34,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:34,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:34,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:34,847 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:55:34,847 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@23fd75fb[Count = 0] remaining members to acquire global barrier 2023-04-16 00:55:34,847 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:55:34,847 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:34,851 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:34,851 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:34,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:34,851 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:55:34,851 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:55:34,851 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:55:34,851 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:34,851 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:55:34,854 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:55:34,854 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:34,854 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:34,854 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:55:34,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:34,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:34,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:34,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:34,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:34,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:34,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:34,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:34,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:34,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:34,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:55:34,857 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:55:34,857 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:55:34,857 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:55:34,857 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:55:34,857 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:55:34,858 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:55:34,859 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:34,859 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:34,859 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:34,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:34,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:34,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:34,859 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:34,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:34,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:34,860 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:34,860 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:34,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:34,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:34,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:34,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:34,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:34,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:34,867 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:34,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:34,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:34,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:34,873 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:34,873 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:34,873 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:34,873 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:34,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:34,873 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:55:34,873 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:55:34,873 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:55:34,873 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:34,873 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:34,873 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:34,873 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:34,873 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:34,873 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:34,873 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:34,873 DEBUG [Thread-2414] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:55:34,873 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:34,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:34,874 DEBUG [Thread-2414] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:55:34,974 DEBUG [Thread-2414] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:55:34,974 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:55:34,976 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:55:34,977 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:55:34,977 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:55:34,977 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:55:34,977 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:34,977 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:55:34,977 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:55:34,978 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:34,978 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:34,979 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:34,979 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:34,979 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:34,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:34,979 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:34,979 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:55:34,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:34,980 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:34,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:55:34,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:34,980 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:55:34,980 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:55:34,980 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:55:34,980 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:55:34,980 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:34,980 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:55:34,981 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:55:34,981 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:55:34,981 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. 2023-04-16 00:55:34,981 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. started... 2023-04-16 00:55:34,981 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. as already flushing 2023-04-16 00:55:34,981 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. 2023-04-16 00:55:34,981 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:55:34,981 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:55:34,981 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:34,981 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:55:34,981 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:55:34,982 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:34,982 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:34,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:34,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:34,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:34,983 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:34,983 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:55:34,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:34,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:34,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:34,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:34,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:34,984 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:55:34,984 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@9a98cfe[Count = 0] remaining members to acquire global barrier 2023-04-16 00:55:34,984 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:55:34,984 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:34,985 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:34,985 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:34,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:34,985 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:55:34,985 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:55:34,985 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:55:34,985 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:34,986 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:55:34,988 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:34,988 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:55:34,988 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:34,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:34,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:34,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:34,988 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:55:34,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:34,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:34,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:34,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:34,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:34,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:34,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:34,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:55:34,990 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:55:34,990 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:55:34,990 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:55:34,990 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:55:34,990 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:55:34,990 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:55:34,992 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:34,992 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:34,992 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:34,992 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:34,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:34,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:34,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:34,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:34,992 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:34,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:34,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:34,993 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:34,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:34,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:34,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:34,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:34,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:34,994 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:34,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:34,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:34,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:34,996 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:34,996 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:34,996 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:34,996 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:34,996 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:55:34,996 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:55:34,996 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:34,996 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:34,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:34,996 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:34,996 DEBUG [Thread-2414] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:55:34,996 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:34,996 DEBUG [Thread-2414] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:55:34,996 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:34,996 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:34,996 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:34,996 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:34,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:35,071 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:35,071 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 233 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59664 deadline: 1681606595069, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:35,096 DEBUG [Thread-2414] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:55:35,097 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:55:35,098 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:55:35,099 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:55:35,099 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:55:35,099 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:55:35,099 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:35,099 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:55:35,100 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:55:35,100 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:35,100 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:35,101 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:35,101 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:35,101 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:35,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:35,101 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:35,101 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:55:35,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:35,102 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:35,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:55:35,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:35,102 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:55:35,102 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:55:35,102 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:55:35,102 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:55:35,102 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:35,102 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:55:35,103 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:55:35,103 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:55:35,103 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. 2023-04-16 00:55:35,103 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. started... 2023-04-16 00:55:35,103 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. as already flushing 2023-04-16 00:55:35,103 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. 2023-04-16 00:55:35,103 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:55:35,103 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:55:35,103 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:35,103 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:55:35,103 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:55:35,105 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:35,105 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:35,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:35,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:35,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:35,105 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:35,106 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:55:35,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:35,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:35,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:35,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:35,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:35,107 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:55:35,107 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@41eee1d0[Count = 0] remaining members to acquire global barrier 2023-04-16 00:55:35,107 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:55:35,107 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:35,108 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:35,108 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:35,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:35,108 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:55:35,108 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:55:35,108 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:55:35,108 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:35,108 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:55:35,109 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:55:35,109 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:35,109 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:35,109 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:55:35,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:35,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:35,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:35,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:35,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:35,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:35,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:35,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:35,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:35,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:35,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:55:35,111 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:55:35,111 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:55:35,111 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:55:35,111 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:55:35,111 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:55:35,111 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:55:35,113 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:35,113 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:35,113 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:35,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:35,113 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:35,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:35,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:35,116 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:35,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:35,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:35,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:35,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:35,116 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:35,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:35,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:35,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:35,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:35,117 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:35,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:35,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:35,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:35,120 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:35,120 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:35,120 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:35,120 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:35,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:35,120 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:35,120 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:55:35,120 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:55:35,120 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:35,120 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:35,121 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:35,121 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:35,121 DEBUG [Thread-2414] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:55:35,121 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:35,121 DEBUG [Thread-2414] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:55:35,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:35,121 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:35,121 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:35,221 DEBUG [Thread-2414] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:55:35,221 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:55:35,223 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:55:35,224 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:55:35,224 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:55:35,224 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:55:35,224 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:35,224 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:55:35,224 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:55:35,225 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:35,225 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:35,226 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:35,226 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:35,226 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:35,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:35,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:35,226 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:35,226 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:55:35,227 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:35,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:55:35,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:35,228 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:55:35,228 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:55:35,228 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:55:35,229 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:55:35,229 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:35,229 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:55:35,229 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:55:35,229 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:55:35,229 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. 2023-04-16 00:55:35,229 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. started... 2023-04-16 00:55:35,229 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. as already flushing 2023-04-16 00:55:35,229 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. 2023-04-16 00:55:35,229 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:55:35,229 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:55:35,229 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:35,229 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:55:35,229 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:55:35,230 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:35,230 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:35,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:35,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:35,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:35,231 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:35,231 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:55:35,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:35,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:35,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:35,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:35,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:35,233 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:55:35,233 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@559f9ec7[Count = 0] remaining members to acquire global barrier 2023-04-16 00:55:35,233 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:55:35,233 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:35,234 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:35,234 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:35,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:35,234 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:55:35,234 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:55:35,234 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:55:35,235 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:35,235 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:55:35,237 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:55:35,237 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:35,237 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:55:35,237 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:35,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:35,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:35,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:35,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:35,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:35,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:35,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:35,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:35,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:35,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:35,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:55:35,240 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:55:35,240 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=510 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/.tmp/C/f30d2a08b43141dcba72337a3f5401ca 2023-04-16 00:55:35,240 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:55:35,240 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:55:35,241 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:55:35,241 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:55:35,241 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:55:35,244 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/.tmp/A/dbcbd7228f01411289b79cc1e02db3c8 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/A/dbcbd7228f01411289b79cc1e02db3c8 2023-04-16 00:55:35,245 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:35,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:35,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:35,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:35,245 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:35,246 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:35,246 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:35,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:35,246 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:35,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:35,246 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:35,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:35,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:35,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:35,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:35,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:35,247 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:35,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:35,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:35,248 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/A/dbcbd7228f01411289b79cc1e02db3c8, entries=200, sequenceid=510, filesize=32.3 K 2023-04-16 00:55:35,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:35,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:35,249 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/.tmp/B/2d0610a124d14e25a02244de2de67474 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/B/2d0610a124d14e25a02244de2de67474 2023-04-16 00:55:35,251 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:35,251 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:35,251 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:35,251 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:35,251 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:35,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:35,251 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:55:35,251 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:55:35,251 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:35,252 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:35,252 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:35,252 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:35,252 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:35,252 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:35,252 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:35,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:35,252 DEBUG [Thread-2414] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:55:35,252 DEBUG [Thread-2414] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:55:35,254 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/B/2d0610a124d14e25a02244de2de67474, entries=150, sequenceid=510, filesize=11.9 K 2023-04-16 00:55:35,254 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/.tmp/C/f30d2a08b43141dcba72337a3f5401ca as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/C/f30d2a08b43141dcba72337a3f5401ca 2023-04-16 00:55:35,259 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/C/f30d2a08b43141dcba72337a3f5401ca, entries=150, sequenceid=510, filesize=11.9 K 2023-04-16 00:55:35,260 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~384.94 KB/394176, currentSize=46.96 KB/48090 for ee2f682ad6dd0fe8e91c021d54d3bdc5 in 861ms, sequenceid=510, compaction requested=true 2023-04-16 00:55:35,260 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for ee2f682ad6dd0fe8e91c021d54d3bdc5: 2023-04-16 00:55:35,260 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 00:55:35,260 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 00:55:35,260 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-16 00:55:35,260 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-16 00:55:35,260 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-16 00:55:35,261 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 78812 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-16 00:55:35,261 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HStore(1912): ee2f682ad6dd0fe8e91c021d54d3bdc5/A is initiating minor compaction (all files) 2023-04-16 00:55:35,261 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of ee2f682ad6dd0fe8e91c021d54d3bdc5/A in TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. 2023-04-16 00:55:35,261 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 34796 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-16 00:55:35,261 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/A/3db9e520b64e4dcb962f0c9eeda597f2, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/A/f0d3276b2571448fa3f2cb37aa8c9b1f, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/A/dbcbd7228f01411289b79cc1e02db3c8] into tmpdir=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/.tmp, totalSize=77.0 K 2023-04-16 00:55:35,261 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HStore(1912): ee2f682ad6dd0fe8e91c021d54d3bdc5/B is initiating minor compaction (all files) 2023-04-16 00:55:35,261 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HRegion(2259): Starting compaction of ee2f682ad6dd0fe8e91c021d54d3bdc5/B in TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. 2023-04-16 00:55:35,261 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/B/390d7fcb0fb14b6f84f6debe382d4c94, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/B/77f861403e0d4883be8ae5d76a678211, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/B/2d0610a124d14e25a02244de2de67474] into tmpdir=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/.tmp, totalSize=34.0 K 2023-04-16 00:55:35,262 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.Compactor(207): Compacting 3db9e520b64e4dcb962f0c9eeda597f2, keycount=150, bloomtype=ROW, size=26.1 K, encoding=NONE, compression=NONE, seqNum=459, earliestPutTs=1681606531766 2023-04-16 00:55:35,262 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting 390d7fcb0fb14b6f84f6debe382d4c94, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=459, earliestPutTs=1681606531766 2023-04-16 00:55:35,262 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.Compactor(207): Compacting f0d3276b2571448fa3f2cb37aa8c9b1f, keycount=100, bloomtype=ROW, size=18.6 K, encoding=NONE, compression=NONE, seqNum=471, earliestPutTs=1681606532112 2023-04-16 00:55:35,262 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting 77f861403e0d4883be8ae5d76a678211, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=471, earliestPutTs=1681606532112 2023-04-16 00:55:35,262 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.Compactor(207): Compacting dbcbd7228f01411289b79cc1e02db3c8, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=510, earliestPutTs=1681606534059 2023-04-16 00:55:35,263 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting 2d0610a124d14e25a02244de2de67474, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=510, earliestPutTs=1681606534075 2023-04-16 00:55:35,268 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] throttle.PressureAwareThroughputController(145): ee2f682ad6dd0fe8e91c021d54d3bdc5#B#compaction#385 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-16 00:55:35,272 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] throttle.PressureAwareThroughputController(145): ee2f682ad6dd0fe8e91c021d54d3bdc5#A#compaction#384 average throughput is 4.93 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-16 00:55:35,352 DEBUG [Thread-2414] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:55:35,353 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:55:35,354 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:55:35,355 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:55:35,355 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:55:35,355 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:55:35,355 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:35,355 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:55:35,355 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:55:35,356 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:35,356 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:35,358 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:35,358 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:35,358 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:35,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:35,358 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:35,358 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:55:35,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:35,358 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:35,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:55:35,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:35,359 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:55:35,359 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:55:35,359 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:55:35,359 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:55:35,359 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:35,359 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:55:35,359 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:55:35,359 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:55:35,360 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. 2023-04-16 00:55:35,360 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. started... 2023-04-16 00:55:35,360 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing ee2f682ad6dd0fe8e91c021d54d3bdc5 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-16 00:55:35,360 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ee2f682ad6dd0fe8e91c021d54d3bdc5, store=A 2023-04-16 00:55:35,360 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:55:35,360 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ee2f682ad6dd0fe8e91c021d54d3bdc5, store=B 2023-04-16 00:55:35,360 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:55:35,360 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ee2f682ad6dd0fe8e91c021d54d3bdc5, store=C 2023-04-16 00:55:35,360 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:55:35,381 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023041697ee571472dd4d1bbb1b68da06013933 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041697ee571472dd4d1bbb1b68da06013933 2023-04-16 00:55:35,586 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(9158): Flush requested on ee2f682ad6dd0fe8e91c021d54d3bdc5 2023-04-16 00:55:35,586 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. as already flushing 2023-04-16 00:55:35,678 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/.tmp/B/f3513bc7064047f9ba257ee6fa526d4e as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/B/f3513bc7064047f9ba257ee6fa526d4e 2023-04-16 00:55:35,685 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in ee2f682ad6dd0fe8e91c021d54d3bdc5/B of ee2f682ad6dd0fe8e91c021d54d3bdc5 into f3513bc7064047f9ba257ee6fa526d4e(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 00:55:35,685 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for ee2f682ad6dd0fe8e91c021d54d3bdc5: 2023-04-16 00:55:35,685 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5., storeName=ee2f682ad6dd0fe8e91c021d54d3bdc5/B, priority=13, startTime=1681606535260; duration=0sec 2023-04-16 00:55:35,685 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 00:55:35,685 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-16 00:55:35,686 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 34796 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-16 00:55:35,686 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HStore(1912): ee2f682ad6dd0fe8e91c021d54d3bdc5/C is initiating minor compaction (all files) 2023-04-16 00:55:35,687 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HRegion(2259): Starting compaction of ee2f682ad6dd0fe8e91c021d54d3bdc5/C in TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. 2023-04-16 00:55:35,687 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/C/0288484dfc034b5eb668ba595be9916c, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/C/08fd47939e6746dda979eb423fc01c8f, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/C/f30d2a08b43141dcba72337a3f5401ca] into tmpdir=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/.tmp, totalSize=34.0 K 2023-04-16 00:55:35,687 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting 0288484dfc034b5eb668ba595be9916c, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=459, earliestPutTs=1681606531766 2023-04-16 00:55:35,688 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting 08fd47939e6746dda979eb423fc01c8f, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=471, earliestPutTs=1681606532112 2023-04-16 00:55:35,688 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting f30d2a08b43141dcba72337a3f5401ca, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=510, earliestPutTs=1681606534075 2023-04-16 00:55:35,689 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/.tmp/A/877f4cae710a4feb967c38f2ac10973d as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/A/877f4cae710a4feb967c38f2ac10973d 2023-04-16 00:55:35,693 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in ee2f682ad6dd0fe8e91c021d54d3bdc5/A of ee2f682ad6dd0fe8e91c021d54d3bdc5 into 877f4cae710a4feb967c38f2ac10973d(size=26.2 K), total size for store is 26.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 00:55:35,693 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for ee2f682ad6dd0fe8e91c021d54d3bdc5: 2023-04-16 00:55:35,693 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5., storeName=ee2f682ad6dd0fe8e91c021d54d3bdc5/A, priority=13, startTime=1681606535260; duration=0sec 2023-04-16 00:55:35,693 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 00:55:35,694 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] throttle.PressureAwareThroughputController(145): ee2f682ad6dd0fe8e91c021d54d3bdc5#C#compaction#387 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-16 00:55:35,714 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681606535681, max timestamp=1681606535712Num uniques 859; 2023-04-16 00:55:35,714 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-16 00:55:35,714 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681606535681, max timestamp=1681606535712Num uniques 859; 2023-04-16 00:55:35,714 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-16 00:55:35,714 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681606535681, max timestamp=1681606535712Num uniques 859; 2023-04-16 00:55:35,714 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-16 00:55:35,787 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=520, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/.tmp/A/59084ab2927743568dee07ebcb621dfe 2023-04-16 00:55:35,821 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:35,821 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 269 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59664 deadline: 1681606595818, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:35,923 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:35,923 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 271 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59664 deadline: 1681606595922, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:36,101 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/.tmp/C/bc82132c03ce439bbda94be65f466c88 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/C/bc82132c03ce439bbda94be65f466c88 2023-04-16 00:55:36,104 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in ee2f682ad6dd0fe8e91c021d54d3bdc5/C of ee2f682ad6dd0fe8e91c021d54d3bdc5 into bc82132c03ce439bbda94be65f466c88(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 00:55:36,105 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for ee2f682ad6dd0fe8e91c021d54d3bdc5: 2023-04-16 00:55:36,105 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5., storeName=ee2f682ad6dd0fe8e91c021d54d3bdc5/C, priority=13, startTime=1681606535260; duration=0sec 2023-04-16 00:55:36,105 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 00:55:36,126 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:36,126 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 273 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59664 deadline: 1681606596125, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:36,202 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=520 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/.tmp/B/96948ea4b56c47b89558bfe9080e16a6 2023-04-16 00:55:36,218 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=520 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/.tmp/C/e1932d000e574a5899aaab73c6587803 2023-04-16 00:55:36,226 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/.tmp/A/59084ab2927743568dee07ebcb621dfe as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/A/59084ab2927743568dee07ebcb621dfe 2023-04-16 00:55:36,231 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/A/59084ab2927743568dee07ebcb621dfe, entries=100, sequenceid=520, filesize=18.6 K 2023-04-16 00:55:36,232 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/.tmp/B/96948ea4b56c47b89558bfe9080e16a6 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/B/96948ea4b56c47b89558bfe9080e16a6 2023-04-16 00:55:36,235 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/B/96948ea4b56c47b89558bfe9080e16a6, entries=100, sequenceid=520, filesize=9.5 K 2023-04-16 00:55:36,236 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/.tmp/C/e1932d000e574a5899aaab73c6587803 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/C/e1932d000e574a5899aaab73c6587803 2023-04-16 00:55:36,239 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/C/e1932d000e574a5899aaab73c6587803, entries=100, sequenceid=520, filesize=9.5 K 2023-04-16 00:55:36,240 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for ee2f682ad6dd0fe8e91c021d54d3bdc5 in 880ms, sequenceid=520, compaction requested=false 2023-04-16 00:55:36,240 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for ee2f682ad6dd0fe8e91c021d54d3bdc5: 2023-04-16 00:55:36,240 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. 2023-04-16 00:55:36,240 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:55:36,240 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:55:36,240 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:36,240 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:55:36,240 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:55:36,242 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:36,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:36,242 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:36,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:36,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:36,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:36,242 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:36,242 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:55:36,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:36,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:36,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:36,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:36,243 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:55:36,243 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1d857b22[Count = 0] remaining members to acquire global barrier 2023-04-16 00:55:36,244 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:55:36,244 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:36,245 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:36,245 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:36,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:36,245 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:36,245 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:55:36,245 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:55:36,245 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:55:36,245 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:55:36,246 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:36,246 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:55:36,247 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:36,247 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:55:36,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:36,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:36,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:36,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:36,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:36,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:36,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:36,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:36,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:36,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:36,249 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(9158): Flush requested on ee2f682ad6dd0fe8e91c021d54d3bdc5 2023-04-16 00:55:36,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:55:36,250 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:55:36,250 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:55:36,250 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:55:36,250 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:55:36,250 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:55:36,250 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:55:36,251 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1681606535779, max timestamp=1681606536248Num uniques 1018; 2023-04-16 00:55:36,251 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-16 00:55:36,251 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1681606535779, max timestamp=1681606536248Num uniques 1018; 2023-04-16 00:55:36,251 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-16 00:55:36,251 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1681606535779, max timestamp=1681606536248Num uniques 1018; 2023-04-16 00:55:36,251 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-16 00:55:36,251 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing ee2f682ad6dd0fe8e91c021d54d3bdc5 3/3 column families, dataSize=26.84 KB heapSize=65.39 KB 2023-04-16 00:55:36,252 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ee2f682ad6dd0fe8e91c021d54d3bdc5, store=A 2023-04-16 00:55:36,252 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 00:55:36,252 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ee2f682ad6dd0fe8e91c021d54d3bdc5, store=B 2023-04-16 00:55:36,252 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 00:55:36,252 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ee2f682ad6dd0fe8e91c021d54d3bdc5, store=C 2023-04-16 00:55:36,252 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 00:55:36,254 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:36,254 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:36,254 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:36,254 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:36,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:36,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:36,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:36,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:36,255 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:36,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:36,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:36,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:36,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:36,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:36,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:36,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:36,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:36,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:36,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:36,264 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:36,265 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:36,266 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304169b14ac54d26445768e76b57690a08f69 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304169b14ac54d26445768e76b57690a08f69 2023-04-16 00:55:36,269 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:36,269 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:36,269 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:36,269 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:36,269 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:36,269 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:36,269 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:36,269 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:36,269 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:36,269 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:36,269 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:55:36,269 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:55:36,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:36,269 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:36,269 DEBUG [Thread-2414] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:55:36,269 DEBUG [Thread-2414] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:55:36,269 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:36,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:36,273 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=561, memsize=8.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/.tmp/A/aad0ed93d2534e72ac353a8fec83844e 2023-04-16 00:55:36,287 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=561 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/.tmp/B/57cc18fe0e1b4138bfac128a1b2b3e50 2023-04-16 00:55:36,304 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681606536288, max timestamp=1681606536302Num uniques 859; 2023-04-16 00:55:36,304 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681606536288, max timestamp=1681606536302Num uniques 859; 2023-04-16 00:55:36,304 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681606536288, max timestamp=1681606536302Num uniques 859; 2023-04-16 00:55:36,304 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-16 00:55:36,304 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-16 00:55:36,305 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-16 00:55:36,313 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=561 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/.tmp/C/34776e19e7684adf860004e20c5aa224 2023-04-16 00:55:36,317 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/.tmp/A/aad0ed93d2534e72ac353a8fec83844e as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/A/aad0ed93d2534e72ac353a8fec83844e 2023-04-16 00:55:36,323 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/A/aad0ed93d2534e72ac353a8fec83844e, entries=200, sequenceid=561, filesize=32.3 K 2023-04-16 00:55:36,323 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/.tmp/B/57cc18fe0e1b4138bfac128a1b2b3e50 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/B/57cc18fe0e1b4138bfac128a1b2b3e50 2023-04-16 00:55:36,326 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/B/57cc18fe0e1b4138bfac128a1b2b3e50, entries=150, sequenceid=561, filesize=11.9 K 2023-04-16 00:55:36,327 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/.tmp/C/34776e19e7684adf860004e20c5aa224 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/C/34776e19e7684adf860004e20c5aa224 2023-04-16 00:55:36,330 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/C/34776e19e7684adf860004e20c5aa224, entries=150, sequenceid=561, filesize=11.9 K 2023-04-16 00:55:36,330 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~65.34 KB/66912, currentSize=93.93 KB/96180 for ee2f682ad6dd0fe8e91c021d54d3bdc5 in 79ms, sequenceid=561, compaction requested=true 2023-04-16 00:55:36,330 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for ee2f682ad6dd0fe8e91c021d54d3bdc5: 2023-04-16 00:55:36,330 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 00:55:36,330 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 00:55:36,330 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-16 00:55:36,330 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-16 00:55:36,330 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 00:55:36,331 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 78914 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-16 00:55:36,331 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 34898 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-16 00:55:36,331 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HStore(1912): ee2f682ad6dd0fe8e91c021d54d3bdc5/A is initiating minor compaction (all files) 2023-04-16 00:55:36,331 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HStore(1912): ee2f682ad6dd0fe8e91c021d54d3bdc5/B is initiating minor compaction (all files) 2023-04-16 00:55:36,331 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of ee2f682ad6dd0fe8e91c021d54d3bdc5/A in TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. 2023-04-16 00:55:36,331 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HRegion(2259): Starting compaction of ee2f682ad6dd0fe8e91c021d54d3bdc5/B in TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. 2023-04-16 00:55:36,331 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/A/877f4cae710a4feb967c38f2ac10973d, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/A/59084ab2927743568dee07ebcb621dfe, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/A/aad0ed93d2534e72ac353a8fec83844e] into tmpdir=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/.tmp, totalSize=77.1 K 2023-04-16 00:55:36,332 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/B/f3513bc7064047f9ba257ee6fa526d4e, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/B/96948ea4b56c47b89558bfe9080e16a6, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/B/57cc18fe0e1b4138bfac128a1b2b3e50] into tmpdir=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/.tmp, totalSize=34.1 K 2023-04-16 00:55:36,332 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.Compactor(207): Compacting 877f4cae710a4feb967c38f2ac10973d, keycount=150, bloomtype=ROW, size=26.2 K, encoding=NONE, compression=NONE, seqNum=510, earliestPutTs=1681606534075 2023-04-16 00:55:36,332 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting f3513bc7064047f9ba257ee6fa526d4e, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=510, earliestPutTs=1681606534075 2023-04-16 00:55:36,332 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.Compactor(207): Compacting 59084ab2927743568dee07ebcb621dfe, keycount=100, bloomtype=ROW, size=18.6 K, encoding=NONE, compression=NONE, seqNum=520, earliestPutTs=1681606534418 2023-04-16 00:55:36,332 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting 96948ea4b56c47b89558bfe9080e16a6, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=520, earliestPutTs=1681606534418 2023-04-16 00:55:36,332 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.Compactor(207): Compacting aad0ed93d2534e72ac353a8fec83844e, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=561, earliestPutTs=1681606535779 2023-04-16 00:55:36,333 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting 57cc18fe0e1b4138bfac128a1b2b3e50, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=561, earliestPutTs=1681606535803 2023-04-16 00:55:36,334 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(9158): Flush requested on ee2f682ad6dd0fe8e91c021d54d3bdc5 2023-04-16 00:55:36,334 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing ee2f682ad6dd0fe8e91c021d54d3bdc5 3/3 column families, dataSize=100.63 KB heapSize=280.69 KB 2023-04-16 00:55:36,334 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ee2f682ad6dd0fe8e91c021d54d3bdc5, store=A 2023-04-16 00:55:36,334 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 00:55:36,334 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ee2f682ad6dd0fe8e91c021d54d3bdc5, store=B 2023-04-16 00:55:36,334 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 00:55:36,335 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ee2f682ad6dd0fe8e91c021d54d3bdc5, store=C 2023-04-16 00:55:36,335 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 00:55:36,339 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] throttle.PressureAwareThroughputController(145): ee2f682ad6dd0fe8e91c021d54d3bdc5#B#compaction#393 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-16 00:55:36,357 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] throttle.PressureAwareThroughputController(145): ee2f682ad6dd0fe8e91c021d54d3bdc5#A#compaction#394 average throughput is 1.10 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-16 00:55:36,360 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/.tmp/B/14b4000f11ae4fbcb0d05afcbccc6008 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/B/14b4000f11ae4fbcb0d05afcbccc6008 2023-04-16 00:55:36,364 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in ee2f682ad6dd0fe8e91c021d54d3bdc5/B of ee2f682ad6dd0fe8e91c021d54d3bdc5 into 14b4000f11ae4fbcb0d05afcbccc6008(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 00:55:36,364 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for ee2f682ad6dd0fe8e91c021d54d3bdc5: 2023-04-16 00:55:36,364 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5., storeName=ee2f682ad6dd0fe8e91c021d54d3bdc5/B, priority=13, startTime=1681606536330; duration=0sec 2023-04-16 00:55:36,364 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 00:55:36,364 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-16 00:55:36,365 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 34898 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-16 00:55:36,365 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HStore(1912): ee2f682ad6dd0fe8e91c021d54d3bdc5/C is initiating minor compaction (all files) 2023-04-16 00:55:36,365 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HRegion(2259): Starting compaction of ee2f682ad6dd0fe8e91c021d54d3bdc5/C in TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. 2023-04-16 00:55:36,365 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/C/bc82132c03ce439bbda94be65f466c88, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/C/e1932d000e574a5899aaab73c6587803, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/C/34776e19e7684adf860004e20c5aa224] into tmpdir=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/.tmp, totalSize=34.1 K 2023-04-16 00:55:36,366 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting bc82132c03ce439bbda94be65f466c88, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=510, earliestPutTs=1681606534075 2023-04-16 00:55:36,366 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting e1932d000e574a5899aaab73c6587803, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=520, earliestPutTs=1681606534418 2023-04-16 00:55:36,366 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting 34776e19e7684adf860004e20c5aa224, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=561, earliestPutTs=1681606535803 2023-04-16 00:55:36,369 DEBUG [Thread-2414] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:55:36,371 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] throttle.PressureAwareThroughputController(145): ee2f682ad6dd0fe8e91c021d54d3bdc5#C#compaction#396 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-16 00:55:36,380 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:55:36,382 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:36,382 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:55:36,382 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 209 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59642 deadline: 1681606596367, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:36,382 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:36,382 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 217 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59650 deadline: 1681606596368, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:36,383 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:36,383 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 208 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59666 deadline: 1681606596369, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:36,384 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:55:36,384 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:55:36,384 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:55:36,384 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:36,384 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:55:36,384 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:55:36,384 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:36,384 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:36,386 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:36,386 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:36,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:36,386 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:36,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:36,386 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/.tmp/C/ba22634f8a894b5a91484b79f77555b6 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/C/ba22634f8a894b5a91484b79f77555b6 2023-04-16 00:55:36,387 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:36,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:55:36,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:36,387 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:55:36,387 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:36,387 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:55:36,392 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:55:36,392 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:55:36,392 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:55:36,392 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:36,392 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:55:36,393 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in ee2f682ad6dd0fe8e91c021d54d3bdc5/C of ee2f682ad6dd0fe8e91c021d54d3bdc5 into ba22634f8a894b5a91484b79f77555b6(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 00:55:36,393 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for ee2f682ad6dd0fe8e91c021d54d3bdc5: 2023-04-16 00:55:36,393 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5., storeName=ee2f682ad6dd0fe8e91c021d54d3bdc5/C, priority=13, startTime=1681606536330; duration=0sec 2023-04-16 00:55:36,393 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 00:55:36,395 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:55:36,395 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:55:36,395 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. 2023-04-16 00:55:36,395 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. started... 2023-04-16 00:55:36,395 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. as already flushing 2023-04-16 00:55:36,395 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. 2023-04-16 00:55:36,395 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:55:36,395 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:55:36,395 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:36,395 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:55:36,395 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:55:36,397 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:36,397 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:36,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:36,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:36,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:36,397 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:36,397 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:55:36,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:36,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:36,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:36,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:36,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:36,401 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:55:36,401 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@588eda10[Count = 0] remaining members to acquire global barrier 2023-04-16 00:55:36,401 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:55:36,401 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:36,404 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:36,404 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:36,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:36,404 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:55:36,404 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:55:36,404 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:55:36,404 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:36,404 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:55:36,405 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:36,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:36,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:36,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:36,405 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:55:36,405 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:36,406 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:55:36,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:36,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:36,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:36,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:36,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:36,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:36,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:36,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:55:36,410 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:55:36,410 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:55:36,410 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:55:36,410 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:55:36,410 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:55:36,410 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:55:36,412 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:36,412 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:36,412 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:36,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:36,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:36,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:36,412 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:36,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:36,412 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:36,412 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:36,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:36,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:36,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:36,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:36,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:36,415 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:36,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:36,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:36,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:36,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:36,418 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:36,418 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:36,418 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:36,418 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:36,418 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:55:36,419 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:55:36,418 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:36,418 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:36,419 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:55:36,418 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:36,418 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:36,419 DEBUG [Thread-2414] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:55:36,419 DEBUG [Thread-2414] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:55:36,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:36,419 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:36,419 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:36,419 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:36,419 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:36,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:36,430 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:36,431 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 275 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59664 deadline: 1681606596429, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:36,489 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:36,489 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 211 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59642 deadline: 1681606596483, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:36,490 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:36,490 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 219 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59650 deadline: 1681606596483, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:36,490 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:36,490 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 210 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59666 deadline: 1681606596484, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:36,519 DEBUG [Thread-2414] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:55:36,519 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:55:36,521 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:55:36,522 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:55:36,522 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:55:36,522 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:55:36,522 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:36,522 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:55:36,522 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:55:36,522 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:36,522 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:36,523 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:36,523 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:36,523 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:36,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:36,523 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:36,523 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:55:36,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:36,524 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:36,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:55:36,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:36,524 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:55:36,524 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:55:36,525 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:55:36,525 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:55:36,525 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:36,525 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:55:36,525 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:55:36,525 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:55:36,525 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. 2023-04-16 00:55:36,525 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. started... 2023-04-16 00:55:36,525 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. as already flushing 2023-04-16 00:55:36,525 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. 2023-04-16 00:55:36,525 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:55:36,525 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:55:36,525 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:36,525 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:55:36,525 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:55:36,527 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:36,527 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:36,527 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:36,527 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:55:36,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:36,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:36,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:36,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:36,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:36,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:36,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:36,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:36,529 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:55:36,529 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6e5850b0[Count = 0] remaining members to acquire global barrier 2023-04-16 00:55:36,529 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:55:36,530 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:36,531 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:36,531 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:36,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:36,531 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:55:36,532 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:55:36,532 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:55:36,533 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:36,533 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:55:36,534 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:55:36,534 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:36,534 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:36,534 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:55:36,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:36,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:36,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:36,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:36,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:36,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:36,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:36,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:36,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:36,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:36,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:55:36,536 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:55:36,536 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:55:36,537 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:55:36,537 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:55:36,537 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:55:36,537 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:55:36,538 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:36,538 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:36,538 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:36,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:36,538 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:36,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:36,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:36,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:36,538 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:36,538 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:36,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:36,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:36,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:36,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:36,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:36,539 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:36,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:36,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:36,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:36,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:36,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:36,545 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:36,545 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:36,545 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:36,545 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:36,545 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:36,545 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:55:36,545 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:55:36,545 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:36,545 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:36,545 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:36,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:36,545 DEBUG [Thread-2414] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:55:36,545 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:36,545 DEBUG [Thread-2414] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:55:36,545 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:55:36,545 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:36,545 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:36,546 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:36,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:36,645 DEBUG [Thread-2414] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:55:36,646 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:55:36,647 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:55:36,648 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:55:36,648 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:55:36,648 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:55:36,648 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:36,648 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:55:36,648 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:55:36,649 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:36,649 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:36,650 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:36,650 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:36,650 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:36,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:36,650 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:36,650 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:55:36,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:36,651 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:36,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:55:36,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:36,651 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:55:36,651 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:55:36,651 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:55:36,651 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:55:36,651 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:36,651 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:55:36,651 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:55:36,651 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:55:36,652 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. 2023-04-16 00:55:36,652 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. started... 2023-04-16 00:55:36,652 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. as already flushing 2023-04-16 00:55:36,652 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. 2023-04-16 00:55:36,652 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:55:36,652 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:55:36,652 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:36,652 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:55:36,652 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:55:36,653 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:36,653 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:36,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:36,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:36,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:36,653 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:36,654 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:55:36,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:36,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:36,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:36,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:36,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:36,655 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:55:36,655 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4718921a[Count = 0] remaining members to acquire global barrier 2023-04-16 00:55:36,655 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:55:36,655 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:36,657 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:36,657 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:36,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:36,657 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:55:36,657 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:55:36,657 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:36,657 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:55:36,657 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:55:36,659 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:36,659 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:55:36,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:36,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:36,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:36,659 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:36,659 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:55:36,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:36,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:36,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:36,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:36,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:36,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:36,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:36,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:55:36,661 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:55:36,661 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:55:36,661 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:55:36,661 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:55:36,661 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:55:36,661 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:55:36,662 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:36,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:36,662 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:36,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:36,663 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:36,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:36,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:36,663 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:36,663 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:36,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:36,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:36,663 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:36,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:36,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:36,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:36,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:36,665 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:36,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:36,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:36,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:36,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:36,667 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:36,667 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:36,667 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:36,667 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:36,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:36,667 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:55:36,667 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:55:36,667 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:36,667 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:36,667 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:36,667 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:36,667 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:36,667 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:36,667 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:36,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:36,667 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:36,667 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:55:36,667 DEBUG [Thread-2414] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:55:36,668 DEBUG [Thread-2414] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:55:36,694 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:36,694 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 213 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59642 deadline: 1681606596690, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:36,694 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:36,694 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 212 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59666 deadline: 1681606596692, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:36,694 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:36,694 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 221 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59650 deadline: 1681606596692, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:36,768 DEBUG [Thread-2414] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:55:36,768 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:55:36,770 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:55:36,772 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:55:36,772 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:55:36,772 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:55:36,772 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:36,772 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:55:36,772 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:55:36,772 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:36,772 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:36,775 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:36,775 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304163d99fed34b774a34b768d4faaf799a20 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304163d99fed34b774a34b768d4faaf799a20 2023-04-16 00:55:36,775 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:36,775 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:55:36,776 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:36,776 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:36,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:36,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:36,776 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:36,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:55:36,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:36,777 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:55:36,777 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:55:36,777 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:55:36,778 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:55:36,778 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:36,778 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:55:36,778 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:55:36,778 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:55:36,778 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. 2023-04-16 00:55:36,778 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. started... 2023-04-16 00:55:36,778 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. as already flushing 2023-04-16 00:55:36,778 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. 2023-04-16 00:55:36,778 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:55:36,778 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:55:36,778 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:36,778 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:55:36,778 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:55:36,780 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:36,780 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:36,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:36,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:36,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:36,780 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:36,781 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:55:36,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:36,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:36,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:36,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:36,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:36,782 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:55:36,782 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@71f2adb0[Count = 0] remaining members to acquire global barrier 2023-04-16 00:55:36,782 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:55:36,782 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:36,783 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:36,783 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:36,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:36,784 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:55:36,784 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:55:36,784 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:55:36,784 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:36,784 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:55:36,785 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:36,785 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:55:36,785 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:36,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:36,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:36,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:36,785 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:55:36,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:36,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:36,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:36,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:36,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:36,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:36,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:36,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:55:36,787 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:55:36,787 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:55:36,787 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:55:36,787 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:55:36,787 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:55:36,787 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:55:36,788 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=592, memsize=33.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/.tmp/A/58d3203f6ce644a3bf65292bc5d916bf 2023-04-16 00:55:36,790 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:36,790 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:36,790 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:36,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:36,790 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:36,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:36,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:36,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:36,791 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:36,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:36,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:36,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:36,793 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:36,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:36,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:36,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:36,794 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:36,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:36,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:36,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:36,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:36,797 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:36,797 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:36,797 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:36,797 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:36,797 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:36,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:36,797 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:36,797 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:36,797 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:36,797 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:36,797 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:36,798 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:36,798 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:55:36,798 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:55:36,798 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:36,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:36,798 DEBUG [Thread-2414] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:55:36,798 DEBUG [Thread-2414] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:55:36,798 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:55:36,825 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=592 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/.tmp/B/477406cf8a034b55bd4dfbf50013a3c4 2023-04-16 00:55:36,832 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/.tmp/A/4ef4f7b33ca24b36b7f45eca269fec1f as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/A/4ef4f7b33ca24b36b7f45eca269fec1f 2023-04-16 00:55:36,846 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in ee2f682ad6dd0fe8e91c021d54d3bdc5/A of ee2f682ad6dd0fe8e91c021d54d3bdc5 into 4ef4f7b33ca24b36b7f45eca269fec1f(size=26.3 K), total size for store is 26.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 00:55:36,846 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for ee2f682ad6dd0fe8e91c021d54d3bdc5: 2023-04-16 00:55:36,846 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5., storeName=ee2f682ad6dd0fe8e91c021d54d3bdc5/A, priority=13, startTime=1681606536330; duration=0sec 2023-04-16 00:55:36,846 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 00:55:36,851 DEBUG [Thread-2419] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x68a0e21a to 127.0.0.1:53150 2023-04-16 00:55:36,851 DEBUG [Thread-2419] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-16 00:55:36,857 DEBUG [Thread-2423] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1d7ba242 to 127.0.0.1:53150 2023-04-16 00:55:36,857 DEBUG [Thread-2423] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-16 00:55:36,858 DEBUG [Thread-2415] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7e01d5e5 to 127.0.0.1:53150 2023-04-16 00:55:36,858 DEBUG [Thread-2415] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-16 00:55:36,862 DEBUG [Thread-2417] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x51ccd8b7 to 127.0.0.1:53150 2023-04-16 00:55:36,862 DEBUG [Thread-2417] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-16 00:55:36,864 DEBUG [Thread-2421] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1ec12b02 to 127.0.0.1:53150 2023-04-16 00:55:36,865 DEBUG [Thread-2421] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-16 00:55:36,898 DEBUG [Thread-2414] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:55:36,899 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:55:36,937 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:36,937 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 277 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59664 deadline: 1681606596937, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:36,996 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:36,996 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 214 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59666 deadline: 1681606596996, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:36,997 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:36,997 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 215 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59642 deadline: 1681606596997, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:36,997 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:36,997 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 223 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59650 deadline: 1681606596997, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:37,241 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=592 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/.tmp/C/34e27ea4e72046beb32a1bc50af70fdc 2023-04-16 00:55:37,244 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/.tmp/A/58d3203f6ce644a3bf65292bc5d916bf as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/A/58d3203f6ce644a3bf65292bc5d916bf 2023-04-16 00:55:37,246 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/A/58d3203f6ce644a3bf65292bc5d916bf, entries=150, sequenceid=592, filesize=25.4 K 2023-04-16 00:55:37,247 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/.tmp/B/477406cf8a034b55bd4dfbf50013a3c4 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/B/477406cf8a034b55bd4dfbf50013a3c4 2023-04-16 00:55:37,249 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/B/477406cf8a034b55bd4dfbf50013a3c4, entries=150, sequenceid=592, filesize=11.9 K 2023-04-16 00:55:37,250 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/.tmp/C/34e27ea4e72046beb32a1bc50af70fdc as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/C/34e27ea4e72046beb32a1bc50af70fdc 2023-04-16 00:55:37,252 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/C/34e27ea4e72046beb32a1bc50af70fdc, entries=150, sequenceid=592, filesize=11.9 K 2023-04-16 00:55:37,254 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~280.64 KB/287376, currentSize=80.51 KB/82440 for ee2f682ad6dd0fe8e91c021d54d3bdc5 in 920ms, sequenceid=592, compaction requested=false 2023-04-16 00:55:37,254 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for ee2f682ad6dd0fe8e91c021d54d3bdc5: 2023-04-16 00:55:37,500 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(9158): Flush requested on ee2f682ad6dd0fe8e91c021d54d3bdc5 2023-04-16 00:55:37,500 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing ee2f682ad6dd0fe8e91c021d54d3bdc5 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-04-16 00:55:37,500 DEBUG [Thread-2404] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1a5ab074 to 127.0.0.1:53150 2023-04-16 00:55:37,500 DEBUG [Thread-2404] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-16 00:55:37,501 DEBUG [Thread-2408] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x38341134 to 127.0.0.1:53150 2023-04-16 00:55:37,501 DEBUG [Thread-2408] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-16 00:55:37,501 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ee2f682ad6dd0fe8e91c021d54d3bdc5, store=A 2023-04-16 00:55:37,501 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:55:37,501 DEBUG [Thread-2410] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x71b5058b to 127.0.0.1:53150 2023-04-16 00:55:37,501 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ee2f682ad6dd0fe8e91c021d54d3bdc5, store=B 2023-04-16 00:55:37,501 DEBUG [Thread-2410] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-16 00:55:37,501 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:55:37,501 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ee2f682ad6dd0fe8e91c021d54d3bdc5, store=C 2023-04-16 00:55:37,501 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:55:37,511 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023041617f0479efad2443da345fd2e96b388a9 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041617f0479efad2443da345fd2e96b388a9 2023-04-16 00:55:37,514 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=613, memsize=33.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/.tmp/A/c7d76a198ef44ee580873da18fb89282 2023-04-16 00:55:37,725 DEBUG [Thread-2406] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x0bcfc1e0 to 127.0.0.1:53150 2023-04-16 00:55:37,725 DEBUG [Thread-2406] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-16 00:55:37,930 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=613 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/.tmp/B/7c1edb6e58dc40e2949eac9aa15da649 2023-04-16 00:55:37,945 DEBUG [Thread-2412] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3620ee35 to 127.0.0.1:53150 2023-04-16 00:55:37,945 DEBUG [Thread-2412] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-16 00:55:37,945 INFO [Listener at localhost/33477] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-16 00:55:37,945 INFO [Listener at localhost/33477] hbase.AcidGuaranteesTestTool(394): wrote 118 2023-04-16 00:55:37,945 INFO [Listener at localhost/33477] hbase.AcidGuaranteesTestTool(394): wrote 19 2023-04-16 00:55:37,945 INFO [Listener at localhost/33477] hbase.AcidGuaranteesTestTool(394): wrote 110 2023-04-16 00:55:37,946 INFO [Listener at localhost/33477] hbase.AcidGuaranteesTestTool(394): wrote 109 2023-04-16 00:55:37,946 INFO [Listener at localhost/33477] hbase.AcidGuaranteesTestTool(394): wrote 164 2023-04-16 00:55:37,946 INFO [Listener at localhost/33477] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-16 00:55:37,946 INFO [Listener at localhost/33477] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-16 00:55:37,946 INFO [Listener at localhost/33477] hbase.AcidGuaranteesTestTool(402): scanned 1561 2023-04-16 00:55:37,946 INFO [Listener at localhost/33477] hbase.AcidGuaranteesTestTool(403): verified 4683 rows 2023-04-16 00:55:37,946 INFO [Listener at localhost/33477] hbase.AcidGuaranteesTestTool(402): scanned 1565 2023-04-16 00:55:37,946 INFO [Listener at localhost/33477] hbase.AcidGuaranteesTestTool(403): verified 4695 rows 2023-04-16 00:55:37,946 INFO [Listener at localhost/33477] hbase.AcidGuaranteesTestTool(402): scanned 1566 2023-04-16 00:55:37,946 INFO [Listener at localhost/33477] hbase.AcidGuaranteesTestTool(403): verified 4698 rows 2023-04-16 00:55:37,946 INFO [Listener at localhost/33477] hbase.AcidGuaranteesTestTool(402): scanned 1568 2023-04-16 00:55:37,946 INFO [Listener at localhost/33477] hbase.AcidGuaranteesTestTool(403): verified 4704 rows 2023-04-16 00:55:37,946 INFO [Listener at localhost/33477] hbase.AcidGuaranteesTestTool(402): scanned 1566 2023-04-16 00:55:37,946 INFO [Listener at localhost/33477] hbase.AcidGuaranteesTestTool(403): verified 4695 rows 2023-04-16 00:55:37,946 INFO [Listener at localhost/33477] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-16 00:55:37,946 DEBUG [Listener at localhost/33477] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x0fd3d54e to 127.0.0.1:53150 2023-04-16 00:55:37,946 DEBUG [Listener at localhost/33477] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-16 00:55:37,950 INFO [Listener at localhost/33477] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-16 00:55:37,950 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable TestAcidGuarantees 2023-04-16 00:55:37,951 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure2.ProcedureExecutor(1029): Stored pid=43, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-16 00:55:37,956 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-16 00:55:37,957 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681606537957"}]},"ts":"1681606537957"} 2023-04-16 00:55:37,958 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-16 00:55:37,960 INFO [PEWorker-5] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-16 00:55:37,960 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=44, ppid=43, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=ee2f682ad6dd0fe8e91c021d54d3bdc5, UNASSIGN}] 2023-04-16 00:55:37,961 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, ppid=43, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=ee2f682ad6dd0fe8e91c021d54d3bdc5, UNASSIGN 2023-04-16 00:55:37,962 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=44 updating hbase:meta row=ee2f682ad6dd0fe8e91c021d54d3bdc5, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:37,962 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681606537962"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681606537962"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681606537962"}]},"ts":"1681606537962"} 2023-04-16 00:55:37,963 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=45, ppid=44, state=RUNNABLE; CloseRegionProcedure ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217}] 2023-04-16 00:55:38,057 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-16 00:55:38,114 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close ee2f682ad6dd0fe8e91c021d54d3bdc5 2023-04-16 00:55:38,114 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing ee2f682ad6dd0fe8e91c021d54d3bdc5, disabling compactions & flushes 2023-04-16 00:55:38,114 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1858): waiting for 0 compactions & cache flush to complete for region TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. 2023-04-16 00:55:38,258 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-16 00:55:38,338 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=613 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/.tmp/C/f36369e71b3a40ed9e1e065bcb9148dc 2023-04-16 00:55:38,341 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/.tmp/A/c7d76a198ef44ee580873da18fb89282 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/A/c7d76a198ef44ee580873da18fb89282 2023-04-16 00:55:38,344 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/A/c7d76a198ef44ee580873da18fb89282, entries=150, sequenceid=613, filesize=25.4 K 2023-04-16 00:55:38,344 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/.tmp/B/7c1edb6e58dc40e2949eac9aa15da649 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/B/7c1edb6e58dc40e2949eac9aa15da649 2023-04-16 00:55:38,347 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/B/7c1edb6e58dc40e2949eac9aa15da649, entries=150, sequenceid=613, filesize=11.9 K 2023-04-16 00:55:38,348 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/.tmp/C/f36369e71b3a40ed9e1e065bcb9148dc as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/C/f36369e71b3a40ed9e1e065bcb9148dc 2023-04-16 00:55:38,350 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/C/f36369e71b3a40ed9e1e065bcb9148dc, entries=150, sequenceid=613, filesize=11.9 K 2023-04-16 00:55:38,351 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=13.42 KB/13740 for ee2f682ad6dd0fe8e91c021d54d3bdc5 in 851ms, sequenceid=613, compaction requested=true 2023-04-16 00:55:38,351 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for ee2f682ad6dd0fe8e91c021d54d3bdc5: 2023-04-16 00:55:38,351 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. 2023-04-16 00:55:38,351 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 00:55:38,351 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. 2023-04-16 00:55:38,351 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. because compaction request was cancelled 2023-04-16 00:55:38,351 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. because compaction request was cancelled 2023-04-16 00:55:38,351 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. after waiting 0 ms 2023-04-16 00:55:38,351 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 00:55:38,351 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. 2023-04-16 00:55:38,351 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. because compaction request was cancelled 2023-04-16 00:55:38,351 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing ee2f682ad6dd0fe8e91c021d54d3bdc5 3/3 column families, dataSize=13.42 KB heapSize=40.59 KB 2023-04-16 00:55:38,351 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 00:55:38,351 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ee2f682ad6dd0fe8e91c021d54d3bdc5, store=A 2023-04-16 00:55:38,352 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:55:38,352 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ee2f682ad6dd0fe8e91c021d54d3bdc5, store=B 2023-04-16 00:55:38,352 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:55:38,352 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ee2f682ad6dd0fe8e91c021d54d3bdc5, store=C 2023-04-16 00:55:38,352 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:55:38,559 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-16 00:55:38,761 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230416f0ae4608bad843eba4f17b32e7bc0b3d to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230416f0ae4608bad843eba4f17b32e7bc0b3d 2023-04-16 00:55:39,059 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-16 00:55:39,165 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=618, memsize=4.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/.tmp/A/42d665fd3d454d6cbbcf9594655310b3 2023-04-16 00:55:39,573 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=4.47 KB at sequenceid=618 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/.tmp/B/76d345ae8b3c430a8387b467e37da34a 2023-04-16 00:55:39,981 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=4.47 KB at sequenceid=618 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/.tmp/C/e324d43058ed4352a6266db4807d9c76 2023-04-16 00:55:39,984 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/.tmp/A/42d665fd3d454d6cbbcf9594655310b3 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/A/42d665fd3d454d6cbbcf9594655310b3 2023-04-16 00:55:39,987 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/A/42d665fd3d454d6cbbcf9594655310b3, entries=50, sequenceid=618, filesize=11.7 K 2023-04-16 00:55:39,987 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/.tmp/B/76d345ae8b3c430a8387b467e37da34a as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/B/76d345ae8b3c430a8387b467e37da34a 2023-04-16 00:55:39,990 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/B/76d345ae8b3c430a8387b467e37da34a, entries=50, sequenceid=618, filesize=7.1 K 2023-04-16 00:55:39,991 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/.tmp/C/e324d43058ed4352a6266db4807d9c76 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/C/e324d43058ed4352a6266db4807d9c76 2023-04-16 00:55:39,994 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/C/e324d43058ed4352a6266db4807d9c76, entries=50, sequenceid=618, filesize=7.1 K 2023-04-16 00:55:39,994 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~13.42 KB/13740, heapSize ~40.55 KB/41520, currentSize=0 B/0 for ee2f682ad6dd0fe8e91c021d54d3bdc5 in 1643ms, sequenceid=618, compaction requested=true 2023-04-16 00:55:40,025 DEBUG [StoreCloser-TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/A/c16608668d934f83821faa9ccec59a9c, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/A/3674f578eab14439be5aa79c8c833cca, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/A/a75b8168cc3348629249af945ef659d6, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/A/37f399b95f1b49f5af8d5a410b18d60f, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/A/2071d651f31a4115bfe4039c27575f60, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/A/2ed4b45335904963b9fb9d185fb956b0, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/A/932b3720baa94283b6e6b897b5ac522f, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/A/182cec2bc8564e7488bff17b333a205f, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/A/678fac82f9f74535baae96803b8563de, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/A/fcb8dc602cda460989697a888a026465, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/A/8ac9840dd9f84421b70650c911dd55c3, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/A/a1de6d30d39b472f8f371b2fe15e3c2a, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/A/fc876d5198cf459d8a9c1daa73d0369c, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/A/71c120ce99bc4fe7ae656fd0cd4f9285, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/A/418be8f08cb8465fbe40a26df541c06b, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/A/08a334d7fa7742b29e578e57fd47127b, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/A/e9f7d4d51d1d45978e62a9839dba11c3, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/A/47203eb368ac4ee58710963f6e345192, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/A/746a2ef59cfc4e9bac430d63a8efb03f, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/A/7232901846364a55b5cf6b7877b7c105, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/A/3db9e520b64e4dcb962f0c9eeda597f2, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/A/f0d3276b2571448fa3f2cb37aa8c9b1f, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/A/dbcbd7228f01411289b79cc1e02db3c8, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/A/877f4cae710a4feb967c38f2ac10973d, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/A/59084ab2927743568dee07ebcb621dfe, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/A/aad0ed93d2534e72ac353a8fec83844e] to archive 2023-04-16 00:55:40,026 DEBUG [StoreCloser-TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-16 00:55:40,027 DEBUG [StoreCloser-TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/A/c16608668d934f83821faa9ccec59a9c to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/A/c16608668d934f83821faa9ccec59a9c 2023-04-16 00:55:40,028 DEBUG [StoreCloser-TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/A/3674f578eab14439be5aa79c8c833cca to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/A/3674f578eab14439be5aa79c8c833cca 2023-04-16 00:55:40,029 DEBUG [StoreCloser-TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/A/a75b8168cc3348629249af945ef659d6 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/A/a75b8168cc3348629249af945ef659d6 2023-04-16 00:55:40,030 DEBUG [StoreCloser-TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/A/37f399b95f1b49f5af8d5a410b18d60f to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/A/37f399b95f1b49f5af8d5a410b18d60f 2023-04-16 00:55:40,030 DEBUG [StoreCloser-TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/A/2071d651f31a4115bfe4039c27575f60 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/A/2071d651f31a4115bfe4039c27575f60 2023-04-16 00:55:40,031 DEBUG [StoreCloser-TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/A/2ed4b45335904963b9fb9d185fb956b0 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/A/2ed4b45335904963b9fb9d185fb956b0 2023-04-16 00:55:40,032 DEBUG [StoreCloser-TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/A/932b3720baa94283b6e6b897b5ac522f to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/A/932b3720baa94283b6e6b897b5ac522f 2023-04-16 00:55:40,033 DEBUG [StoreCloser-TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/A/182cec2bc8564e7488bff17b333a205f to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/A/182cec2bc8564e7488bff17b333a205f 2023-04-16 00:55:40,034 DEBUG [StoreCloser-TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/A/678fac82f9f74535baae96803b8563de to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/A/678fac82f9f74535baae96803b8563de 2023-04-16 00:55:40,035 DEBUG [StoreCloser-TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/A/fcb8dc602cda460989697a888a026465 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/A/fcb8dc602cda460989697a888a026465 2023-04-16 00:55:40,036 DEBUG [StoreCloser-TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/A/8ac9840dd9f84421b70650c911dd55c3 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/A/8ac9840dd9f84421b70650c911dd55c3 2023-04-16 00:55:40,036 DEBUG [StoreCloser-TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/A/a1de6d30d39b472f8f371b2fe15e3c2a to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/A/a1de6d30d39b472f8f371b2fe15e3c2a 2023-04-16 00:55:40,037 DEBUG [StoreCloser-TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/A/fc876d5198cf459d8a9c1daa73d0369c to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/A/fc876d5198cf459d8a9c1daa73d0369c 2023-04-16 00:55:40,038 DEBUG [StoreCloser-TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/A/71c120ce99bc4fe7ae656fd0cd4f9285 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/A/71c120ce99bc4fe7ae656fd0cd4f9285 2023-04-16 00:55:40,039 DEBUG [StoreCloser-TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/A/418be8f08cb8465fbe40a26df541c06b to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/A/418be8f08cb8465fbe40a26df541c06b 2023-04-16 00:55:40,039 DEBUG [StoreCloser-TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/A/08a334d7fa7742b29e578e57fd47127b to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/A/08a334d7fa7742b29e578e57fd47127b 2023-04-16 00:55:40,040 DEBUG [StoreCloser-TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/A/e9f7d4d51d1d45978e62a9839dba11c3 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/A/e9f7d4d51d1d45978e62a9839dba11c3 2023-04-16 00:55:40,041 DEBUG [StoreCloser-TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/A/47203eb368ac4ee58710963f6e345192 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/A/47203eb368ac4ee58710963f6e345192 2023-04-16 00:55:40,042 DEBUG [StoreCloser-TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/A/746a2ef59cfc4e9bac430d63a8efb03f to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/A/746a2ef59cfc4e9bac430d63a8efb03f 2023-04-16 00:55:40,043 DEBUG [StoreCloser-TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/A/7232901846364a55b5cf6b7877b7c105 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/A/7232901846364a55b5cf6b7877b7c105 2023-04-16 00:55:40,044 DEBUG [StoreCloser-TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/A/3db9e520b64e4dcb962f0c9eeda597f2 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/A/3db9e520b64e4dcb962f0c9eeda597f2 2023-04-16 00:55:40,045 DEBUG [StoreCloser-TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/A/f0d3276b2571448fa3f2cb37aa8c9b1f to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/A/f0d3276b2571448fa3f2cb37aa8c9b1f 2023-04-16 00:55:40,046 DEBUG [StoreCloser-TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/A/dbcbd7228f01411289b79cc1e02db3c8 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/A/dbcbd7228f01411289b79cc1e02db3c8 2023-04-16 00:55:40,047 DEBUG [StoreCloser-TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/A/877f4cae710a4feb967c38f2ac10973d to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/A/877f4cae710a4feb967c38f2ac10973d 2023-04-16 00:55:40,048 DEBUG [StoreCloser-TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/A/59084ab2927743568dee07ebcb621dfe to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/A/59084ab2927743568dee07ebcb621dfe 2023-04-16 00:55:40,049 DEBUG [StoreCloser-TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/A/aad0ed93d2534e72ac353a8fec83844e to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/A/aad0ed93d2534e72ac353a8fec83844e 2023-04-16 00:55:40,051 DEBUG [StoreCloser-TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/B/3ee45c1680e74c349af7a713cdd7d5bb, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/B/ce17b32b4d9743d79dbce98b90ba3f29, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/B/fe881913baab42d4ac296653925cddba, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/B/c9a7176dd8534721976f4dbc1076206d, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/B/2b0f80526d1b4839b99cb342be941119, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/B/b73a63612e3049c4b22bb00e99791539, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/B/054b59a0405246e8b8017056a77f2352, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/B/7d50407fb9fe4c1e902f8ded55f85a01, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/B/2c02ceba33224305bed1757852beaca3, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/B/67320ea88b954a078c0608b9799b3e08, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/B/578d5bb5951a4745b077be16e0997cfa, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/B/e5729e88d4db4396a45a9ba5a4709ecf, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/B/41512abe23d34f36b66e488ca81523d1, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/B/e21f9ae5cfbf4cbeb56266c98cbfab59, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/B/0fdda349dad24a0982930e05a8ce1053, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/B/7f59ab69e3be495aa277833d85b29246, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/B/529f111b0dd2430387ad55fd414e7016, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/B/2883ee63702c42a2bc36dc63953f2384, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/B/7b8ed50205274804ba5f8e96c9da8fa2, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/B/390d7fcb0fb14b6f84f6debe382d4c94, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/B/1858cee89d46435da7e61e248a57e0ac, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/B/77f861403e0d4883be8ae5d76a678211, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/B/f3513bc7064047f9ba257ee6fa526d4e, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/B/2d0610a124d14e25a02244de2de67474, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/B/96948ea4b56c47b89558bfe9080e16a6, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/B/57cc18fe0e1b4138bfac128a1b2b3e50] to archive 2023-04-16 00:55:40,052 DEBUG [StoreCloser-TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-16 00:55:40,053 DEBUG [StoreCloser-TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/B/3ee45c1680e74c349af7a713cdd7d5bb to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/B/3ee45c1680e74c349af7a713cdd7d5bb 2023-04-16 00:55:40,054 DEBUG [StoreCloser-TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/B/ce17b32b4d9743d79dbce98b90ba3f29 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/B/ce17b32b4d9743d79dbce98b90ba3f29 2023-04-16 00:55:40,055 DEBUG [StoreCloser-TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/B/fe881913baab42d4ac296653925cddba to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/B/fe881913baab42d4ac296653925cddba 2023-04-16 00:55:40,056 DEBUG [StoreCloser-TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/B/c9a7176dd8534721976f4dbc1076206d to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/B/c9a7176dd8534721976f4dbc1076206d 2023-04-16 00:55:40,057 DEBUG [StoreCloser-TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/B/2b0f80526d1b4839b99cb342be941119 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/B/2b0f80526d1b4839b99cb342be941119 2023-04-16 00:55:40,058 DEBUG [StoreCloser-TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/B/b73a63612e3049c4b22bb00e99791539 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/B/b73a63612e3049c4b22bb00e99791539 2023-04-16 00:55:40,059 DEBUG [StoreCloser-TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/B/054b59a0405246e8b8017056a77f2352 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/B/054b59a0405246e8b8017056a77f2352 2023-04-16 00:55:40,060 DEBUG [StoreCloser-TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/B/7d50407fb9fe4c1e902f8ded55f85a01 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/B/7d50407fb9fe4c1e902f8ded55f85a01 2023-04-16 00:55:40,060 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-16 00:55:40,061 DEBUG [StoreCloser-TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/B/2c02ceba33224305bed1757852beaca3 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/B/2c02ceba33224305bed1757852beaca3 2023-04-16 00:55:40,061 DEBUG [StoreCloser-TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/B/67320ea88b954a078c0608b9799b3e08 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/B/67320ea88b954a078c0608b9799b3e08 2023-04-16 00:55:40,062 DEBUG [StoreCloser-TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/B/578d5bb5951a4745b077be16e0997cfa to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/B/578d5bb5951a4745b077be16e0997cfa 2023-04-16 00:55:40,064 DEBUG [StoreCloser-TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/B/e5729e88d4db4396a45a9ba5a4709ecf to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/B/e5729e88d4db4396a45a9ba5a4709ecf 2023-04-16 00:55:40,065 DEBUG [StoreCloser-TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/B/41512abe23d34f36b66e488ca81523d1 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/B/41512abe23d34f36b66e488ca81523d1 2023-04-16 00:55:40,066 DEBUG [StoreCloser-TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/B/e21f9ae5cfbf4cbeb56266c98cbfab59 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/B/e21f9ae5cfbf4cbeb56266c98cbfab59 2023-04-16 00:55:40,067 DEBUG [StoreCloser-TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/B/0fdda349dad24a0982930e05a8ce1053 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/B/0fdda349dad24a0982930e05a8ce1053 2023-04-16 00:55:40,068 DEBUG [StoreCloser-TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/B/7f59ab69e3be495aa277833d85b29246 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/B/7f59ab69e3be495aa277833d85b29246 2023-04-16 00:55:40,069 DEBUG [StoreCloser-TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/B/529f111b0dd2430387ad55fd414e7016 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/B/529f111b0dd2430387ad55fd414e7016 2023-04-16 00:55:40,070 DEBUG [StoreCloser-TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/B/2883ee63702c42a2bc36dc63953f2384 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/B/2883ee63702c42a2bc36dc63953f2384 2023-04-16 00:55:40,071 DEBUG [StoreCloser-TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/B/7b8ed50205274804ba5f8e96c9da8fa2 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/B/7b8ed50205274804ba5f8e96c9da8fa2 2023-04-16 00:55:40,072 DEBUG [StoreCloser-TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/B/390d7fcb0fb14b6f84f6debe382d4c94 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/B/390d7fcb0fb14b6f84f6debe382d4c94 2023-04-16 00:55:40,074 DEBUG [StoreCloser-TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/B/1858cee89d46435da7e61e248a57e0ac to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/B/1858cee89d46435da7e61e248a57e0ac 2023-04-16 00:55:40,075 DEBUG [StoreCloser-TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/B/77f861403e0d4883be8ae5d76a678211 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/B/77f861403e0d4883be8ae5d76a678211 2023-04-16 00:55:40,075 DEBUG [StoreCloser-TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/B/f3513bc7064047f9ba257ee6fa526d4e to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/B/f3513bc7064047f9ba257ee6fa526d4e 2023-04-16 00:55:40,077 DEBUG [StoreCloser-TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/B/2d0610a124d14e25a02244de2de67474 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/B/2d0610a124d14e25a02244de2de67474 2023-04-16 00:55:40,078 DEBUG [StoreCloser-TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/B/96948ea4b56c47b89558bfe9080e16a6 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/B/96948ea4b56c47b89558bfe9080e16a6 2023-04-16 00:55:40,079 DEBUG [StoreCloser-TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/B/57cc18fe0e1b4138bfac128a1b2b3e50 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/B/57cc18fe0e1b4138bfac128a1b2b3e50 2023-04-16 00:55:40,082 DEBUG [StoreCloser-TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/C/71299a44e639499e8a3034bde1608367, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/C/fbdd963920664561b5cbe6b2544fc6c3, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/C/dee497a3a72947de84df13174cb2a796, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/C/9e1ac9a463654bab82cbfd048ba6294f, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/C/ada9cb30cf754d93858849098089397e, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/C/5cd6c75d70c8467c9df564e03dd709f2, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/C/49b65699a7b04bf291d28f7195af0c98, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/C/f78762c94569489a86c3607be4af2a47, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/C/29c995d0bad14ddeb5ac50be29be5566, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/C/0065cc7bde58460e98227dcbec7c11cc, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/C/89572d68c50148d89d56409b4eab3f9b, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/C/667ca7c094594c9eb7a4a5fbb6614e26, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/C/a59c947f064844b0b361c5e65f5f3869, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/C/3294108abd13461d82dfb6cd003b584b, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/C/157ff7be7222483aa909be17f0f6304b, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/C/6bfa232107b046459a24b7697d4116ec, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/C/ab710e19efc841b298189bda8852f1ee, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/C/ee7334928c3641a68884204f77c716e2, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/C/8b02f65560944dc5884bc589421eec3a, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/C/0288484dfc034b5eb668ba595be9916c, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/C/ff4d879155a94de0a34ace3c7b7def23, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/C/08fd47939e6746dda979eb423fc01c8f, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/C/bc82132c03ce439bbda94be65f466c88, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/C/f30d2a08b43141dcba72337a3f5401ca, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/C/e1932d000e574a5899aaab73c6587803, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/C/34776e19e7684adf860004e20c5aa224] to archive 2023-04-16 00:55:40,082 DEBUG [StoreCloser-TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-16 00:55:40,083 DEBUG [StoreCloser-TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/C/71299a44e639499e8a3034bde1608367 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/C/71299a44e639499e8a3034bde1608367 2023-04-16 00:55:40,084 DEBUG [StoreCloser-TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/C/fbdd963920664561b5cbe6b2544fc6c3 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/C/fbdd963920664561b5cbe6b2544fc6c3 2023-04-16 00:55:40,085 DEBUG [StoreCloser-TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/C/dee497a3a72947de84df13174cb2a796 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/C/dee497a3a72947de84df13174cb2a796 2023-04-16 00:55:40,086 DEBUG [StoreCloser-TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/C/9e1ac9a463654bab82cbfd048ba6294f to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/C/9e1ac9a463654bab82cbfd048ba6294f 2023-04-16 00:55:40,087 DEBUG [StoreCloser-TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/C/ada9cb30cf754d93858849098089397e to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/C/ada9cb30cf754d93858849098089397e 2023-04-16 00:55:40,088 DEBUG [StoreCloser-TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/C/5cd6c75d70c8467c9df564e03dd709f2 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/C/5cd6c75d70c8467c9df564e03dd709f2 2023-04-16 00:55:40,089 DEBUG [StoreCloser-TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/C/49b65699a7b04bf291d28f7195af0c98 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/C/49b65699a7b04bf291d28f7195af0c98 2023-04-16 00:55:40,089 DEBUG [StoreCloser-TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/C/f78762c94569489a86c3607be4af2a47 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/C/f78762c94569489a86c3607be4af2a47 2023-04-16 00:55:40,090 DEBUG [StoreCloser-TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/C/29c995d0bad14ddeb5ac50be29be5566 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/C/29c995d0bad14ddeb5ac50be29be5566 2023-04-16 00:55:40,091 DEBUG [StoreCloser-TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/C/0065cc7bde58460e98227dcbec7c11cc to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/C/0065cc7bde58460e98227dcbec7c11cc 2023-04-16 00:55:40,092 DEBUG [StoreCloser-TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/C/89572d68c50148d89d56409b4eab3f9b to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/C/89572d68c50148d89d56409b4eab3f9b 2023-04-16 00:55:40,093 DEBUG [StoreCloser-TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/C/667ca7c094594c9eb7a4a5fbb6614e26 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/C/667ca7c094594c9eb7a4a5fbb6614e26 2023-04-16 00:55:40,094 DEBUG [StoreCloser-TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/C/a59c947f064844b0b361c5e65f5f3869 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/C/a59c947f064844b0b361c5e65f5f3869 2023-04-16 00:55:40,094 DEBUG [StoreCloser-TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/C/3294108abd13461d82dfb6cd003b584b to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/C/3294108abd13461d82dfb6cd003b584b 2023-04-16 00:55:40,095 DEBUG [StoreCloser-TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/C/157ff7be7222483aa909be17f0f6304b to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/C/157ff7be7222483aa909be17f0f6304b 2023-04-16 00:55:40,096 DEBUG [StoreCloser-TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/C/6bfa232107b046459a24b7697d4116ec to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/C/6bfa232107b046459a24b7697d4116ec 2023-04-16 00:55:40,097 DEBUG [StoreCloser-TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/C/ab710e19efc841b298189bda8852f1ee to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/C/ab710e19efc841b298189bda8852f1ee 2023-04-16 00:55:40,101 DEBUG [StoreCloser-TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/C/ee7334928c3641a68884204f77c716e2 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/C/ee7334928c3641a68884204f77c716e2 2023-04-16 00:55:40,102 DEBUG [StoreCloser-TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/C/8b02f65560944dc5884bc589421eec3a to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/C/8b02f65560944dc5884bc589421eec3a 2023-04-16 00:55:40,103 DEBUG [StoreCloser-TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/C/0288484dfc034b5eb668ba595be9916c to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/C/0288484dfc034b5eb668ba595be9916c 2023-04-16 00:55:40,104 DEBUG [StoreCloser-TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/C/ff4d879155a94de0a34ace3c7b7def23 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/C/ff4d879155a94de0a34ace3c7b7def23 2023-04-16 00:55:40,105 DEBUG [StoreCloser-TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/C/08fd47939e6746dda979eb423fc01c8f to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/C/08fd47939e6746dda979eb423fc01c8f 2023-04-16 00:55:40,105 DEBUG [StoreCloser-TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/C/bc82132c03ce439bbda94be65f466c88 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/C/bc82132c03ce439bbda94be65f466c88 2023-04-16 00:55:40,106 DEBUG [StoreCloser-TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/C/f30d2a08b43141dcba72337a3f5401ca to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/C/f30d2a08b43141dcba72337a3f5401ca 2023-04-16 00:55:40,107 DEBUG [StoreCloser-TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/C/e1932d000e574a5899aaab73c6587803 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/C/e1932d000e574a5899aaab73c6587803 2023-04-16 00:55:40,108 DEBUG [StoreCloser-TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/C/34776e19e7684adf860004e20c5aa224 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/C/34776e19e7684adf860004e20c5aa224 2023-04-16 00:55:40,111 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/recovered.edits/621.seqid, newMaxSeqId=621, maxSeqId=4 2023-04-16 00:55:40,112 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5. 2023-04-16 00:55:40,112 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for ee2f682ad6dd0fe8e91c021d54d3bdc5: 2023-04-16 00:55:40,113 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed ee2f682ad6dd0fe8e91c021d54d3bdc5 2023-04-16 00:55:40,114 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=44 updating hbase:meta row=ee2f682ad6dd0fe8e91c021d54d3bdc5, regionState=CLOSED 2023-04-16 00:55:40,114 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681606540114"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681606540114"}]},"ts":"1681606540114"} 2023-04-16 00:55:40,116 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=45, resume processing ppid=44 2023-04-16 00:55:40,116 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=45, ppid=44, state=SUCCESS; CloseRegionProcedure ee2f682ad6dd0fe8e91c021d54d3bdc5, server=jenkins-hbase4.apache.org,34975,1681606419217 in 2.1520 sec 2023-04-16 00:55:40,117 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=44, resume processing ppid=43 2023-04-16 00:55:40,117 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=44, ppid=43, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=ee2f682ad6dd0fe8e91c021d54d3bdc5, UNASSIGN in 2.1560 sec 2023-04-16 00:55:40,118 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681606540118"}]},"ts":"1681606540118"} 2023-04-16 00:55:40,119 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-16 00:55:40,120 INFO [PEWorker-5] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-16 00:55:40,121 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=43, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 2.1690 sec 2023-04-16 00:55:40,557 DEBUG [regionserver/jenkins-hbase4:0.Chore.1] throttle.PressureAwareCompactionThroughputController(103): CompactionPressure is 0.0, tune throughput to 50.00 MB/second 2023-04-16 00:55:42,061 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-16 00:55:42,061 INFO [Listener at localhost/33477] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 43 completed 2023-04-16 00:55:42,062 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete TestAcidGuarantees 2023-04-16 00:55:42,063 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure2.ProcedureExecutor(1029): Stored pid=46, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-16 00:55:42,065 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=46, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-16 00:55:42,065 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=46, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-16 00:55:42,065 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1227): Checking to see if procedure is done pid=46 2023-04-16 00:55:42,069 DEBUG [HFileArchiver-9] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/.tmp/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5 2023-04-16 00:55:42,071 DEBUG [HFileArchiver-9] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/.tmp/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/A, FileablePath, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/.tmp/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/B, FileablePath, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/.tmp/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/C, FileablePath, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/.tmp/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/recovered.edits] 2023-04-16 00:55:42,074 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/.tmp/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/A/42d665fd3d454d6cbbcf9594655310b3 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/A/42d665fd3d454d6cbbcf9594655310b3 2023-04-16 00:55:42,075 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/.tmp/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/A/4ef4f7b33ca24b36b7f45eca269fec1f to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/A/4ef4f7b33ca24b36b7f45eca269fec1f 2023-04-16 00:55:42,076 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/.tmp/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/A/58d3203f6ce644a3bf65292bc5d916bf to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/A/58d3203f6ce644a3bf65292bc5d916bf 2023-04-16 00:55:42,078 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/.tmp/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/A/c7d76a198ef44ee580873da18fb89282 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/A/c7d76a198ef44ee580873da18fb89282 2023-04-16 00:55:42,080 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/.tmp/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/B/14b4000f11ae4fbcb0d05afcbccc6008 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/B/14b4000f11ae4fbcb0d05afcbccc6008 2023-04-16 00:55:42,081 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/.tmp/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/B/477406cf8a034b55bd4dfbf50013a3c4 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/B/477406cf8a034b55bd4dfbf50013a3c4 2023-04-16 00:55:42,089 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/.tmp/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/B/76d345ae8b3c430a8387b467e37da34a to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/B/76d345ae8b3c430a8387b467e37da34a 2023-04-16 00:55:42,091 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/.tmp/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/B/7c1edb6e58dc40e2949eac9aa15da649 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/B/7c1edb6e58dc40e2949eac9aa15da649 2023-04-16 00:55:42,092 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/.tmp/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/C/34e27ea4e72046beb32a1bc50af70fdc to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/C/34e27ea4e72046beb32a1bc50af70fdc 2023-04-16 00:55:42,093 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/.tmp/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/C/ba22634f8a894b5a91484b79f77555b6 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/C/ba22634f8a894b5a91484b79f77555b6 2023-04-16 00:55:42,094 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/.tmp/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/C/e324d43058ed4352a6266db4807d9c76 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/C/e324d43058ed4352a6266db4807d9c76 2023-04-16 00:55:42,095 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/.tmp/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/C/f36369e71b3a40ed9e1e065bcb9148dc to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/C/f36369e71b3a40ed9e1e065bcb9148dc 2023-04-16 00:55:42,097 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/.tmp/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/recovered.edits/621.seqid to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5/recovered.edits/621.seqid 2023-04-16 00:55:42,098 DEBUG [HFileArchiver-9] backup.HFileArchiver(596): Deleted hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/.tmp/data/default/TestAcidGuarantees/ee2f682ad6dd0fe8e91c021d54d3bdc5 2023-04-16 00:55:42,098 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-16 00:55:42,098 DEBUG [PEWorker-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-16 00:55:42,099 DEBUG [PEWorker-1] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A] 2023-04-16 00:55:42,101 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041606bdb7a505f64291b3e8bf3e8642aa97 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041606bdb7a505f64291b3e8bf3e8642aa97 2023-04-16 00:55:42,102 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041617f0479efad2443da345fd2e96b388a9 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041617f0479efad2443da345fd2e96b388a9 2023-04-16 00:55:42,103 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230416223d6d34841f4bb7a437f41ee93e8f70 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230416223d6d34841f4bb7a437f41ee93e8f70 2023-04-16 00:55:42,104 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041636e9ea1f89ab49ca81c501edaf5ca24c to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041636e9ea1f89ab49ca81c501edaf5ca24c 2023-04-16 00:55:42,105 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304163b31b2f0f7b246f9837c73691e3c3c00 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304163b31b2f0f7b246f9837c73691e3c3c00 2023-04-16 00:55:42,106 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304163d99fed34b774a34b768d4faaf799a20 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304163d99fed34b774a34b768d4faaf799a20 2023-04-16 00:55:42,106 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304163e6d6685152f4c508d0b57ecaebca119 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304163e6d6685152f4c508d0b57ecaebca119 2023-04-16 00:55:42,107 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230416405eb547e55c4c88afa4f0414c569e03 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230416405eb547e55c4c88afa4f0414c569e03 2023-04-16 00:55:42,108 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304165efeb500254e48aeb84708afe2a65db2 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304165efeb500254e48aeb84708afe2a65db2 2023-04-16 00:55:42,109 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304166f9c830f47b94e1ebf27375c688fd445 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304166f9c830f47b94e1ebf27375c688fd445 2023-04-16 00:55:42,110 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041671c540bf6462428aaefbfb07db9cab0d to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041671c540bf6462428aaefbfb07db9cab0d 2023-04-16 00:55:42,111 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041677904b95d71b4d72b4cd7fbfd3678e80 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041677904b95d71b4d72b4cd7fbfd3678e80 2023-04-16 00:55:42,112 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304167b9587ae906a4ff5849a7b83aafc68ba to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304167b9587ae906a4ff5849a7b83aafc68ba 2023-04-16 00:55:42,113 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041682d926f025344c3cac852cd356a261e0 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041682d926f025344c3cac852cd356a261e0 2023-04-16 00:55:42,114 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041695f2c39d277243af86e1f816526a4ef0 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041695f2c39d277243af86e1f816526a4ef0 2023-04-16 00:55:42,115 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041697ee571472dd4d1bbb1b68da06013933 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041697ee571472dd4d1bbb1b68da06013933 2023-04-16 00:55:42,115 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304169b14ac54d26445768e76b57690a08f69 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304169b14ac54d26445768e76b57690a08f69 2023-04-16 00:55:42,116 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230416a0bf3deb697a4343b03479b66f1b327c to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230416a0bf3deb697a4343b03479b66f1b327c 2023-04-16 00:55:42,117 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230416bc17d0e51c5c45c2a2b357d4447c3cfd to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230416bc17d0e51c5c45c2a2b357d4447c3cfd 2023-04-16 00:55:42,118 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230416c00629f1e82e49db975e698aa16e4542 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230416c00629f1e82e49db975e698aa16e4542 2023-04-16 00:55:42,119 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230416d6f389e251784292816936f8a45e25d1 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230416d6f389e251784292816936f8a45e25d1 2023-04-16 00:55:42,120 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230416df80ceb84edf48f48ec6fbce63c178d5 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230416df80ceb84edf48f48ec6fbce63c178d5 2023-04-16 00:55:42,121 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230416f0ae4608bad843eba4f17b32e7bc0b3d to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230416f0ae4608bad843eba4f17b32e7bc0b3d 2023-04-16 00:55:42,122 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230416f102adec4ffa43448e7bf5f94191795e to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230416f102adec4ffa43448e7bf5f94191795e 2023-04-16 00:55:42,122 DEBUG [PEWorker-1] backup.HFileArchiver(596): Deleted hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-16 00:55:42,124 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=46, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-16 00:55:42,126 WARN [PEWorker-1] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-16 00:55:42,127 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-16 00:55:42,128 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=46, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-16 00:55:42,128 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-16 00:55:42,128 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1681606542128"}]},"ts":"9223372036854775807"} 2023-04-16 00:55:42,129 INFO [PEWorker-1] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-16 00:55:42,129 DEBUG [PEWorker-1] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => ee2f682ad6dd0fe8e91c021d54d3bdc5, NAME => 'TestAcidGuarantees,,1681606513775.ee2f682ad6dd0fe8e91c021d54d3bdc5.', STARTKEY => '', ENDKEY => ''}] 2023-04-16 00:55:42,129 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-16 00:55:42,130 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1681606542130"}]},"ts":"9223372036854775807"} 2023-04-16 00:55:42,131 INFO [PEWorker-1] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-16 00:55:42,132 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(130): Finished pid=46, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-16 00:55:42,133 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=46, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 70 msec 2023-04-16 00:55:42,166 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1227): Checking to see if procedure is done pid=46 2023-04-16 00:55:42,166 INFO [Listener at localhost/33477] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 46 completed 2023-04-16 00:55:42,178 INFO [Listener at localhost/33477] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testMobScanAtomicity Thread=228 (was 225) - Thread LEAK? -, OpenFileDescriptor=568 (was 543) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=396 (was 305) - SystemLoadAverage LEAK? -, ProcessCount=171 (was 171), AvailableMemoryMB=1438 (was 752) - AvailableMemoryMB LEAK? - 2023-04-16 00:55:42,189 INFO [Listener at localhost/33477] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testScanAtomicity Thread=228, OpenFileDescriptor=568, MaxFileDescriptor=60000, SystemLoadAverage=396, ProcessCount=171, AvailableMemoryMB=1438 2023-04-16 00:55:42,191 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-16 00:55:42,191 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-16 00:55:42,192 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure2.ProcedureExecutor(1029): Stored pid=47, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-16 00:55:42,193 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-16 00:55:42,193 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 47 2023-04-16 00:55:42,194 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-16 00:55:42,194 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-04-16 00:55:42,195 DEBUG [HFileArchiver-10] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/.tmp/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c 2023-04-16 00:55:42,195 DEBUG [HFileArchiver-10] backup.HFileArchiver(153): Directory hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/.tmp/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c empty. 2023-04-16 00:55:42,196 DEBUG [HFileArchiver-10] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/.tmp/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c 2023-04-16 00:55:42,196 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-16 00:55:42,207 DEBUG [PEWorker-3] util.FSTableDescriptors(570): Wrote into hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-16 00:55:42,208 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => 0abd14932d4ec4e5fdc6177f2c7cc55c, NAME => 'TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/.tmp 2023-04-16 00:55:42,222 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-16 00:55:42,222 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing 0abd14932d4ec4e5fdc6177f2c7cc55c, disabling compactions & flushes 2023-04-16 00:55:42,222 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. 2023-04-16 00:55:42,222 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. 2023-04-16 00:55:42,222 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. after waiting 0 ms 2023-04-16 00:55:42,222 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. 2023-04-16 00:55:42,222 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. 2023-04-16 00:55:42,222 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for 0abd14932d4ec4e5fdc6177f2c7cc55c: 2023-04-16 00:55:42,224 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-16 00:55:42,225 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681606542225"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681606542225"}]},"ts":"1681606542225"} 2023-04-16 00:55:42,226 INFO [PEWorker-3] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-16 00:55:42,227 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-16 00:55:42,227 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681606542227"}]},"ts":"1681606542227"} 2023-04-16 00:55:42,228 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-16 00:55:42,232 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=48, ppid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=0abd14932d4ec4e5fdc6177f2c7cc55c, ASSIGN}] 2023-04-16 00:55:42,233 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=48, ppid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=0abd14932d4ec4e5fdc6177f2c7cc55c, ASSIGN 2023-04-16 00:55:42,234 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=48, ppid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=0abd14932d4ec4e5fdc6177f2c7cc55c, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,34975,1681606419217; forceNewPlan=false, retain=false 2023-04-16 00:55:42,295 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-04-16 00:55:42,385 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=48 updating hbase:meta row=0abd14932d4ec4e5fdc6177f2c7cc55c, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:42,385 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681606542385"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681606542385"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681606542385"}]},"ts":"1681606542385"} 2023-04-16 00:55:42,386 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=49, ppid=48, state=RUNNABLE; OpenRegionProcedure 0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217}] 2023-04-16 00:55:42,496 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-04-16 00:55:42,542 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. 2023-04-16 00:55:42,542 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 0abd14932d4ec4e5fdc6177f2c7cc55c, NAME => 'TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c.', STARTKEY => '', ENDKEY => ''} 2023-04-16 00:55:42,542 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 0abd14932d4ec4e5fdc6177f2c7cc55c 2023-04-16 00:55:42,542 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-16 00:55:42,543 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 0abd14932d4ec4e5fdc6177f2c7cc55c 2023-04-16 00:55:42,543 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 0abd14932d4ec4e5fdc6177f2c7cc55c 2023-04-16 00:55:42,544 INFO [StoreOpener-0abd14932d4ec4e5fdc6177f2c7cc55c-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 0abd14932d4ec4e5fdc6177f2c7cc55c 2023-04-16 00:55:42,546 DEBUG [StoreOpener-0abd14932d4ec4e5fdc6177f2c7cc55c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/A 2023-04-16 00:55:42,546 DEBUG [StoreOpener-0abd14932d4ec4e5fdc6177f2c7cc55c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/A 2023-04-16 00:55:42,546 INFO [StoreOpener-0abd14932d4ec4e5fdc6177f2c7cc55c-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-16 00:55:42,546 INFO [StoreOpener-0abd14932d4ec4e5fdc6177f2c7cc55c-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 0abd14932d4ec4e5fdc6177f2c7cc55c columnFamilyName A 2023-04-16 00:55:42,547 INFO [StoreOpener-0abd14932d4ec4e5fdc6177f2c7cc55c-1] regionserver.HStore(310): Store=0abd14932d4ec4e5fdc6177f2c7cc55c/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-16 00:55:42,547 INFO [StoreOpener-0abd14932d4ec4e5fdc6177f2c7cc55c-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 0abd14932d4ec4e5fdc6177f2c7cc55c 2023-04-16 00:55:42,548 DEBUG [StoreOpener-0abd14932d4ec4e5fdc6177f2c7cc55c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/B 2023-04-16 00:55:42,548 DEBUG [StoreOpener-0abd14932d4ec4e5fdc6177f2c7cc55c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/B 2023-04-16 00:55:42,549 INFO [StoreOpener-0abd14932d4ec4e5fdc6177f2c7cc55c-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-16 00:55:42,549 INFO [StoreOpener-0abd14932d4ec4e5fdc6177f2c7cc55c-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 0abd14932d4ec4e5fdc6177f2c7cc55c columnFamilyName B 2023-04-16 00:55:42,549 INFO [StoreOpener-0abd14932d4ec4e5fdc6177f2c7cc55c-1] regionserver.HStore(310): Store=0abd14932d4ec4e5fdc6177f2c7cc55c/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-16 00:55:42,549 INFO [StoreOpener-0abd14932d4ec4e5fdc6177f2c7cc55c-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 0abd14932d4ec4e5fdc6177f2c7cc55c 2023-04-16 00:55:42,550 DEBUG [StoreOpener-0abd14932d4ec4e5fdc6177f2c7cc55c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/C 2023-04-16 00:55:42,551 DEBUG [StoreOpener-0abd14932d4ec4e5fdc6177f2c7cc55c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/C 2023-04-16 00:55:42,551 INFO [StoreOpener-0abd14932d4ec4e5fdc6177f2c7cc55c-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-16 00:55:42,551 INFO [StoreOpener-0abd14932d4ec4e5fdc6177f2c7cc55c-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 0abd14932d4ec4e5fdc6177f2c7cc55c columnFamilyName C 2023-04-16 00:55:42,551 INFO [StoreOpener-0abd14932d4ec4e5fdc6177f2c7cc55c-1] regionserver.HStore(310): Store=0abd14932d4ec4e5fdc6177f2c7cc55c/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-16 00:55:42,551 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. 2023-04-16 00:55:42,552 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c 2023-04-16 00:55:42,552 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c 2023-04-16 00:55:42,554 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-16 00:55:42,556 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 0abd14932d4ec4e5fdc6177f2c7cc55c 2023-04-16 00:55:42,557 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-16 00:55:42,558 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 0abd14932d4ec4e5fdc6177f2c7cc55c; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=62431321, jitterRate=-0.06970082223415375}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-16 00:55:42,558 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 0abd14932d4ec4e5fdc6177f2c7cc55c: 2023-04-16 00:55:42,558 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c., pid=49, masterSystemTime=1681606542538 2023-04-16 00:55:42,560 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. 2023-04-16 00:55:42,560 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. 2023-04-16 00:55:42,560 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=48 updating hbase:meta row=0abd14932d4ec4e5fdc6177f2c7cc55c, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:42,560 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681606542560"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1681606542560"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1681606542560"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1681606542560"}]},"ts":"1681606542560"} 2023-04-16 00:55:42,563 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=49, resume processing ppid=48 2023-04-16 00:55:42,563 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=49, ppid=48, state=SUCCESS; OpenRegionProcedure 0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 in 175 msec 2023-04-16 00:55:42,564 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=48, resume processing ppid=47 2023-04-16 00:55:42,564 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=48, ppid=47, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=0abd14932d4ec4e5fdc6177f2c7cc55c, ASSIGN in 331 msec 2023-04-16 00:55:42,565 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-16 00:55:42,565 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681606542565"}]},"ts":"1681606542565"} 2023-04-16 00:55:42,566 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-16 00:55:42,568 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-16 00:55:42,569 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=47, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 377 msec 2023-04-16 00:55:42,797 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-04-16 00:55:42,797 INFO [Listener at localhost/33477] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 47 completed 2023-04-16 00:55:42,798 DEBUG [Listener at localhost/33477] zookeeper.ReadOnlyZKClient(139): Connect 0x5e372785 to 127.0.0.1:53150 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-16 00:55:42,813 DEBUG [Listener at localhost/33477] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2a987907, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-16 00:55:42,814 DEBUG [Listener at localhost/33477] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-16 00:55:42,817 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:57186, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-16 00:55:42,818 DEBUG [Listener at localhost/33477] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-16 00:55:42,825 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:43130, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-16 00:55:42,826 DEBUG [Listener at localhost/33477] zookeeper.ReadOnlyZKClient(139): Connect 0x501c5a88 to 127.0.0.1:53150 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-16 00:55:42,835 DEBUG [Listener at localhost/33477] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@37705401, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-16 00:55:42,835 DEBUG [Listener at localhost/33477] zookeeper.ReadOnlyZKClient(139): Connect 0x27f2d997 to 127.0.0.1:53150 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-16 00:55:42,842 DEBUG [Listener at localhost/33477] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@15880ffb, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-16 00:55:42,842 DEBUG [Listener at localhost/33477] zookeeper.ReadOnlyZKClient(139): Connect 0x34ad4aa8 to 127.0.0.1:53150 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-16 00:55:42,850 DEBUG [Listener at localhost/33477] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@52203cb, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-16 00:55:42,851 DEBUG [Listener at localhost/33477] zookeeper.ReadOnlyZKClient(139): Connect 0x1c29757d to 127.0.0.1:53150 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-16 00:55:42,856 DEBUG [Listener at localhost/33477] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@420f945a, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-16 00:55:42,856 DEBUG [Listener at localhost/33477] zookeeper.ReadOnlyZKClient(139): Connect 0x06edd017 to 127.0.0.1:53150 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-16 00:55:42,863 DEBUG [Listener at localhost/33477] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@51b8df2a, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-16 00:55:42,864 DEBUG [Listener at localhost/33477] zookeeper.ReadOnlyZKClient(139): Connect 0x67346c47 to 127.0.0.1:53150 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-16 00:55:42,868 DEBUG [Listener at localhost/33477] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@70e2be04, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-16 00:55:42,869 DEBUG [Listener at localhost/33477] zookeeper.ReadOnlyZKClient(139): Connect 0x426cb82a to 127.0.0.1:53150 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-16 00:55:42,875 DEBUG [Listener at localhost/33477] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@67ea274, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-16 00:55:42,876 DEBUG [Listener at localhost/33477] zookeeper.ReadOnlyZKClient(139): Connect 0x5f5363d8 to 127.0.0.1:53150 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-16 00:55:42,884 DEBUG [Listener at localhost/33477] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@78592df0, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-16 00:55:42,885 DEBUG [Listener at localhost/33477] zookeeper.ReadOnlyZKClient(139): Connect 0x211582f2 to 127.0.0.1:53150 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-16 00:55:42,890 DEBUG [Listener at localhost/33477] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@659c68ab, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-16 00:55:42,891 DEBUG [Listener at localhost/33477] zookeeper.ReadOnlyZKClient(139): Connect 0x65816b6e to 127.0.0.1:53150 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-16 00:55:42,900 DEBUG [Listener at localhost/33477] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@680e773e, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-16 00:55:42,918 DEBUG [hconnection-0xf4a6575-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-16 00:55:42,919 DEBUG [hconnection-0x5fc8b720-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-16 00:55:42,919 DEBUG [hconnection-0x27780af8-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-16 00:55:42,920 DEBUG [hconnection-0x5256bb65-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-16 00:55:42,928 DEBUG [hconnection-0x79897226-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-16 00:55:42,928 DEBUG [hconnection-0x2d0f678d-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-16 00:55:42,924 DEBUG [hconnection-0x6174fe6a-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-16 00:55:42,922 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:55:42,931 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:57228, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-16 00:55:42,931 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:57212, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-16 00:55:42,932 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:57248, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-16 00:55:42,921 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:57202, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-16 00:55:42,931 DEBUG [hconnection-0x7b319654-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-16 00:55:42,928 DEBUG [hconnection-0x454ada37-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-16 00:55:42,933 DEBUG [hconnection-0x43d0b66-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-16 00:55:42,936 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:57274, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-16 00:55:42,937 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:57232, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-16 00:55:42,937 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:57268, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-16 00:55:42,936 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:57254, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-16 00:55:42,946 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:57300, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-16 00:55:42,946 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:57288, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-16 00:55:42,964 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:55:42,964 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(9158): Flush requested on 0abd14932d4ec4e5fdc6177f2c7cc55c 2023-04-16 00:55:42,968 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 0abd14932d4ec4e5fdc6177f2c7cc55c 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-16 00:55:42,964 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:55:42,972 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:55:42,972 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:42,972 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:55:42,972 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:55:42,973 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:42,973 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:42,974 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0abd14932d4ec4e5fdc6177f2c7cc55c, store=A 2023-04-16 00:55:42,974 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:55:42,974 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0abd14932d4ec4e5fdc6177f2c7cc55c, store=B 2023-04-16 00:55:42,974 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:55:42,974 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0abd14932d4ec4e5fdc6177f2c7cc55c, store=C 2023-04-16 00:55:42,974 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:55:42,976 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:42,976 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:42,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:42,976 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:42,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:42,977 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:42,977 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:55:42,977 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:42,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:55:42,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:42,977 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:55:42,989 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:55:42,989 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:55:42,989 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:55:42,989 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:42,989 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:55:42,989 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:55:42,989 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:55:42,989 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. 2023-04-16 00:55:42,989 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. started... 2023-04-16 00:55:42,989 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. as already flushing 2023-04-16 00:55:42,989 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. 2023-04-16 00:55:42,989 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:55:42,989 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:55:42,989 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:42,989 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:55:42,989 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:55:42,992 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:42,992 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:42,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:42,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:42,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:42,992 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:42,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:42,992 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:55:42,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:42,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:42,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:42,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:42,993 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:55:42,993 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5f82b7c4[Count = 0] remaining members to acquire global barrier 2023-04-16 00:55:42,993 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:55:42,993 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:42,994 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:42,994 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:42,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:42,995 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:55:42,995 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:55:42,995 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:55:42,995 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:42,995 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:55:42,996 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:42,996 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:55:42,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:42,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:42,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:42,996 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:42,996 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:55:42,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:42,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:42,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:42,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:42,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:42,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:42,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:42,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:55:42,998 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:55:42,998 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:55:42,998 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:55:42,998 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:55:42,998 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:55:42,998 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:55:42,999 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:42,999 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:42,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:42,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:43,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:42,999 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:43,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:42,999 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:43,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:43,000 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:43,000 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:43,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:43,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:43,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:43,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:43,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:43,001 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:43,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:43,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:43,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:43,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:43,004 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:43,004 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:43,004 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:43,004 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:43,004 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:43,004 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:43,004 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:43,004 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:55:43,004 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:55:43,004 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:55:43,004 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:43,004 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:43,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:43,004 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:43,005 DEBUG [Thread-3183] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:55:43,005 DEBUG [Thread-3183] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:55:43,005 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:43,005 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:43,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:43,006 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=14 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp/A/71418beef49441f398ffb9fdee497861 2023-04-16 00:55:43,026 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:43,026 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57212 deadline: 1681606603023, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:43,026 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:43,026 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:43,026 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 5 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57254 deadline: 1681606603025, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:43,026 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57202 deadline: 1681606603025, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:43,027 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:43,027 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:43,027 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57268 deadline: 1681606603026, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:43,027 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 6 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57228 deadline: 1681606603026, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:43,105 DEBUG [Thread-3183] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:55:43,105 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:55:43,107 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:55:43,117 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:55:43,117 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:55:43,117 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:55:43,117 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:43,117 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:55:43,117 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:55:43,117 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:43,117 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:43,119 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:43,119 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:43,119 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:55:43,120 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:43,120 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:43,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:43,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:43,125 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:43,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:55:43,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:43,126 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:55:43,131 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:43,131 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57212 deadline: 1681606603127, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:43,131 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:43,131 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57254 deadline: 1681606603127, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:43,132 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:43,132 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57228 deadline: 1681606603128, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:43,132 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:43,132 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57268 deadline: 1681606603128, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:43,132 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:43,133 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57202 deadline: 1681606603128, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:43,137 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:55:43,137 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:55:43,137 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:55:43,137 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:43,137 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:55:43,139 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:55:43,139 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:55:43,140 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. 2023-04-16 00:55:43,140 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. started... 2023-04-16 00:55:43,140 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. as already flushing 2023-04-16 00:55:43,140 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. 2023-04-16 00:55:43,140 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:55:43,140 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:55:43,140 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:43,147 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:55:43,147 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:55:43,149 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:43,149 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:43,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:43,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:43,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:43,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:43,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:43,150 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:43,150 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:55:43,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:43,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:43,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:43,151 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:55:43,151 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@73f3a154[Count = 0] remaining members to acquire global barrier 2023-04-16 00:55:43,151 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:55:43,151 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:43,156 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:43,156 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:43,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:43,156 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:55:43,156 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:55:43,156 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:55:43,157 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:43,158 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:55:43,160 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:55:43,160 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:43,160 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:43,160 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:55:43,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:43,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:43,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:43,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:43,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:43,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:43,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:43,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:43,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:43,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:43,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:55:43,163 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:55:43,163 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:55:43,163 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:55:43,163 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:55:43,163 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:55:43,163 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:55:43,165 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:43,165 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:43,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:43,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:43,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:43,165 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:43,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:43,165 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:43,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:43,165 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:43,165 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:43,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:43,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:43,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:43,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:43,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:43,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:43,167 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:43,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:43,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:43,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:43,169 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:43,169 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:43,169 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:43,169 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:43,169 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:43,169 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:43,169 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:43,174 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:43,174 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:55:43,174 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:55:43,174 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:43,174 DEBUG [Thread-3183] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:55:43,174 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:43,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:43,174 DEBUG [Thread-3183] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:55:43,174 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:43,175 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:43,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:43,275 DEBUG [Thread-3183] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:55:43,275 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:55:43,278 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:55:43,293 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:55:43,293 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:55:43,293 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:55:43,293 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:43,293 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:55:43,293 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:55:43,293 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:43,293 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:43,295 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:43,295 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:43,295 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:43,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:43,295 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:43,295 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:55:43,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:43,295 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:43,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:55:43,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:43,296 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:55:43,300 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:55:43,300 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:55:43,300 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:55:43,300 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:43,300 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:55:43,310 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:55:43,310 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:55:43,310 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. 2023-04-16 00:55:43,310 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. started... 2023-04-16 00:55:43,310 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. as already flushing 2023-04-16 00:55:43,310 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. 2023-04-16 00:55:43,310 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:55:43,310 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:55:43,310 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:43,313 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:55:43,313 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:55:43,317 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:43,317 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:43,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:43,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:43,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:43,317 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:43,317 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:55:43,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:43,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:43,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:43,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:43,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:43,318 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:55:43,318 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:55:43,318 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@70f86572[Count = 0] remaining members to acquire global barrier 2023-04-16 00:55:43,318 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:43,320 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:43,320 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:55:43,320 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:43,320 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:43,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:43,320 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:55:43,320 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:55:43,320 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:55:43,321 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:43,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:43,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:43,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:43,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:43,322 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:55:43,322 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:43,322 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:55:43,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:43,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:43,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:43,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:43,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:43,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:43,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:55:43,324 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:55:43,324 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:55:43,324 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:55:43,324 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:55:43,324 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:55:43,324 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:55:43,326 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:43,326 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:43,326 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:43,326 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:43,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:43,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:43,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:43,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:43,326 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:43,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:43,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:43,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:43,328 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:43,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:43,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:43,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:43,329 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:43,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:43,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:43,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:43,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:43,333 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:43,334 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57228 deadline: 1681606603333, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:43,334 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:43,334 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57268 deadline: 1681606603333, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:43,335 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:43,336 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57212 deadline: 1681606603335, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:43,336 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:43,337 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57254 deadline: 1681606603336, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:43,337 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:43,337 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57202 deadline: 1681606603336, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:43,339 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:43,339 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:43,339 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:43,339 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:43,339 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:43,339 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:43,339 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:43,339 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:55:43,339 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:55:43,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:43,339 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:43,339 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:43,339 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:55:43,340 DEBUG [Thread-3183] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:55:43,340 DEBUG [Thread-3183] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:55:43,340 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:43,340 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:43,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:43,340 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:43,440 DEBUG [Thread-3183] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:55:43,441 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:55:43,442 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:55:43,451 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:55:43,451 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:55:43,451 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:55:43,451 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:43,451 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:55:43,451 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:55:43,452 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:43,452 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:43,453 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:43,453 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:43,453 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:43,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:43,454 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:43,454 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:55:43,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:43,454 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:43,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:55:43,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:43,455 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:55:43,455 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:55:43,455 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:55:43,455 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:55:43,455 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:43,455 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:55:43,455 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:55:43,455 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:55:43,455 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. 2023-04-16 00:55:43,456 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. started... 2023-04-16 00:55:43,456 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. as already flushing 2023-04-16 00:55:43,456 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. 2023-04-16 00:55:43,456 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:55:43,456 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:55:43,456 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:43,456 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:55:43,456 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:55:43,457 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:43,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:43,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:43,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:43,460 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:43,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:43,462 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=14 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp/B/f4aad6d467e94184902a4a8943a66c2d 2023-04-16 00:55:43,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:43,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:43,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:43,463 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:43,463 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:55:43,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:43,465 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:55:43,465 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1403c9f0[Count = 0] remaining members to acquire global barrier 2023-04-16 00:55:43,465 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:55:43,465 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:43,467 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:43,467 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:43,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:43,467 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:55:43,467 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:55:43,467 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:55:43,467 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:43,467 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:55:43,470 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:43,470 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:55:43,470 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:43,470 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:55:43,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:43,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:43,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:43,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:43,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:43,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:43,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:43,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:43,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:43,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:43,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:55:43,474 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:55:43,474 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:55:43,474 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:55:43,474 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:55:43,475 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:55:43,475 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:55:43,476 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:43,476 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:43,476 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:43,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:43,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:43,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:43,477 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:43,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:43,478 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:43,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:43,478 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:43,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:43,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:43,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:43,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:43,480 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:43,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:43,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:43,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:43,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:43,482 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:43,482 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:43,482 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:43,482 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:43,482 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:43,482 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:55:43,482 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:55:43,482 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:43,482 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:43,482 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:43,482 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:43,482 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:43,482 DEBUG [Thread-3183] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:55:43,482 DEBUG [Thread-3183] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:55:43,482 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:43,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:43,483 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:43,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:43,491 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:55:43,494 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=14 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp/C/a73b04d521c54866abefb4e0094431e3 2023-04-16 00:55:43,500 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp/A/71418beef49441f398ffb9fdee497861 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/A/71418beef49441f398ffb9fdee497861 2023-04-16 00:55:43,506 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/A/71418beef49441f398ffb9fdee497861, entries=200, sequenceid=14, filesize=13.9 K 2023-04-16 00:55:43,506 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp/B/f4aad6d467e94184902a4a8943a66c2d as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/B/f4aad6d467e94184902a4a8943a66c2d 2023-04-16 00:55:43,512 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/B/f4aad6d467e94184902a4a8943a66c2d, entries=150, sequenceid=14, filesize=11.6 K 2023-04-16 00:55:43,514 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp/C/a73b04d521c54866abefb4e0094431e3 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/C/a73b04d521c54866abefb4e0094431e3 2023-04-16 00:55:43,517 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/C/a73b04d521c54866abefb4e0094431e3, entries=150, sequenceid=14, filesize=11.6 K 2023-04-16 00:55:43,518 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=114.05 KB/116790 for 0abd14932d4ec4e5fdc6177f2c7cc55c in 550ms, sequenceid=14, compaction requested=false 2023-04-16 00:55:43,518 DEBUG [MemStoreFlusher.0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'TestAcidGuarantees' 2023-04-16 00:55:43,519 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 0abd14932d4ec4e5fdc6177f2c7cc55c: 2023-04-16 00:55:43,582 DEBUG [Thread-3183] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:55:43,583 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:55:43,586 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:55:43,600 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:55:43,600 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:55:43,600 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:55:43,600 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:43,601 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:55:43,601 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:55:43,601 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:43,601 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:43,602 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:43,602 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:43,602 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:43,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:43,603 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:43,603 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:55:43,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:43,603 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:43,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:55:43,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:43,604 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:55:43,606 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:55:43,606 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:55:43,606 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:55:43,606 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:43,606 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:55:43,606 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:55:43,606 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:55:43,606 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. 2023-04-16 00:55:43,606 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. started... 2023-04-16 00:55:43,606 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 0abd14932d4ec4e5fdc6177f2c7cc55c 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-16 00:55:43,606 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0abd14932d4ec4e5fdc6177f2c7cc55c, store=A 2023-04-16 00:55:43,607 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:55:43,607 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0abd14932d4ec4e5fdc6177f2c7cc55c, store=B 2023-04-16 00:55:43,607 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:55:43,607 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0abd14932d4ec4e5fdc6177f2c7cc55c, store=C 2023-04-16 00:55:43,607 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:55:43,639 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. as already flushing 2023-04-16 00:55:43,640 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(9158): Flush requested on 0abd14932d4ec4e5fdc6177f2c7cc55c 2023-04-16 00:55:43,678 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:43,678 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57202 deadline: 1681606603668, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:43,678 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:43,678 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57228 deadline: 1681606603669, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:43,678 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:43,678 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57212 deadline: 1681606603669, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:43,678 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:43,679 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57268 deadline: 1681606603670, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:43,679 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:43,679 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57254 deadline: 1681606603671, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:43,782 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:43,782 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57202 deadline: 1681606603779, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:43,782 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:43,782 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57228 deadline: 1681606603779, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:43,783 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:43,783 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57212 deadline: 1681606603779, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:43,783 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:43,783 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57268 deadline: 1681606603780, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:43,784 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:43,784 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57254 deadline: 1681606603780, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:43,985 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:43,985 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57228 deadline: 1681606603984, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:43,987 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:43,987 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57268 deadline: 1681606603984, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:43,991 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:43,991 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57212 deadline: 1681606603985, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:43,991 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:43,992 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57202 deadline: 1681606603986, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:43,992 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:43,992 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57254 deadline: 1681606603986, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:44,036 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=34 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp/A/f7a21da1edc6453a87700f88d4768b2c 2023-04-16 00:55:44,049 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=34 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp/B/2446ae0deb3c4113b943f0f34e7e4b9a 2023-04-16 00:55:44,294 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:44,294 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57228 deadline: 1681606604286, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:44,296 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:44,296 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57268 deadline: 1681606604290, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:44,297 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:44,297 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57254 deadline: 1681606604293, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:44,297 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:44,298 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57212 deadline: 1681606604293, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:44,298 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:44,298 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57202 deadline: 1681606604294, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:44,462 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=34 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp/C/330a03ff6a924906b546600756cc25ea 2023-04-16 00:55:44,466 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp/A/f7a21da1edc6453a87700f88d4768b2c as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/A/f7a21da1edc6453a87700f88d4768b2c 2023-04-16 00:55:44,470 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/A/f7a21da1edc6453a87700f88d4768b2c, entries=150, sequenceid=34, filesize=11.6 K 2023-04-16 00:55:44,470 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp/B/2446ae0deb3c4113b943f0f34e7e4b9a as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/B/2446ae0deb3c4113b943f0f34e7e4b9a 2023-04-16 00:55:44,478 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/B/2446ae0deb3c4113b943f0f34e7e4b9a, entries=150, sequenceid=34, filesize=11.6 K 2023-04-16 00:55:44,478 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp/C/330a03ff6a924906b546600756cc25ea as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/C/330a03ff6a924906b546600756cc25ea 2023-04-16 00:55:44,482 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/C/330a03ff6a924906b546600756cc25ea, entries=150, sequenceid=34, filesize=11.6 K 2023-04-16 00:55:44,483 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for 0abd14932d4ec4e5fdc6177f2c7cc55c in 876ms, sequenceid=34, compaction requested=false 2023-04-16 00:55:44,485 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 0abd14932d4ec4e5fdc6177f2c7cc55c: 2023-04-16 00:55:44,485 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. 2023-04-16 00:55:44,489 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:55:44,489 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:55:44,489 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:44,489 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:55:44,489 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:55:44,492 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:44,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:44,492 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:44,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:44,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:44,492 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:44,492 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:55:44,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:44,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:44,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:44,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:44,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:44,494 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:55:44,494 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5ef57023[Count = 0] remaining members to acquire global barrier 2023-04-16 00:55:44,494 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:55:44,494 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:44,495 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:44,495 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:44,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:44,495 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:55:44,495 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:55:44,495 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:55:44,495 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:44,495 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:55:44,497 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:55:44,497 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:44,497 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:55:44,497 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:44,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:44,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:44,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:44,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:44,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:44,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:44,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:44,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:44,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:44,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:44,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:55:44,499 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:55:44,499 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:55:44,499 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:55:44,499 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:55:44,499 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:55:44,499 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:55:44,501 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:44,501 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:44,501 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:44,501 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:44,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:44,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:44,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:44,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:44,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:44,502 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:44,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:44,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:44,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:44,503 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:44,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:44,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:44,503 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:44,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:44,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:44,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:44,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:44,507 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:44,507 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:44,507 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:44,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:44,507 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:44,508 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:44,508 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:44,508 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:44,508 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:44,508 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:44,508 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:44,508 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:44,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:44,508 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:44,508 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:55:44,509 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:55:44,510 DEBUG [Thread-3183] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:55:44,510 DEBUG [Thread-3183] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:55:44,610 DEBUG [Thread-3183] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:55:44,610 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:55:44,612 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:55:44,613 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:55:44,613 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:55:44,613 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:55:44,613 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:44,613 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:55:44,613 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:55:44,614 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:44,614 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:44,615 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:44,615 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:44,615 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:44,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:44,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:44,616 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:44,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:55:44,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:44,616 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:55:44,616 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:44,616 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:55:44,616 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:55:44,616 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:55:44,617 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:55:44,617 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:44,617 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:55:44,617 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:55:44,617 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:55:44,617 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. 2023-04-16 00:55:44,617 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. started... 2023-04-16 00:55:44,618 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 0abd14932d4ec4e5fdc6177f2c7cc55c 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-16 00:55:44,619 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0abd14932d4ec4e5fdc6177f2c7cc55c, store=A 2023-04-16 00:55:44,619 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:55:44,619 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0abd14932d4ec4e5fdc6177f2c7cc55c, store=B 2023-04-16 00:55:44,619 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:55:44,619 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0abd14932d4ec4e5fdc6177f2c7cc55c, store=C 2023-04-16 00:55:44,619 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:55:44,637 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=46 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp/A/a536085eda52409db66262bfc6d27349 2023-04-16 00:55:44,662 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=46 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp/B/87dac584dc4248e28d98db785d1b2d4e 2023-04-16 00:55:44,689 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=46 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp/C/eae239871d0845019a296960b5d4fd40 2023-04-16 00:55:44,693 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp/A/a536085eda52409db66262bfc6d27349 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/A/a536085eda52409db66262bfc6d27349 2023-04-16 00:55:44,697 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/A/a536085eda52409db66262bfc6d27349, entries=150, sequenceid=46, filesize=11.6 K 2023-04-16 00:55:44,697 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp/B/87dac584dc4248e28d98db785d1b2d4e as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/B/87dac584dc4248e28d98db785d1b2d4e 2023-04-16 00:55:44,706 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/B/87dac584dc4248e28d98db785d1b2d4e, entries=150, sequenceid=46, filesize=11.6 K 2023-04-16 00:55:44,707 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp/C/eae239871d0845019a296960b5d4fd40 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/C/eae239871d0845019a296960b5d4fd40 2023-04-16 00:55:44,712 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/C/eae239871d0845019a296960b5d4fd40, entries=150, sequenceid=46, filesize=11.6 K 2023-04-16 00:55:44,713 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=0 B/0 for 0abd14932d4ec4e5fdc6177f2c7cc55c in 95ms, sequenceid=46, compaction requested=true 2023-04-16 00:55:44,713 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 0abd14932d4ec4e5fdc6177f2c7cc55c: 2023-04-16 00:55:44,714 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. 2023-04-16 00:55:44,715 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:55:44,715 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:55:44,715 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:44,715 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:55:44,715 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:55:44,717 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:44,717 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:44,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:44,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:44,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:44,718 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:44,718 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:55:44,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:44,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:44,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:44,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:44,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:44,729 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:55:44,729 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@30e283a8[Count = 0] remaining members to acquire global barrier 2023-04-16 00:55:44,729 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:55:44,729 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:44,731 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:44,732 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:44,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:44,732 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:55:44,732 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:55:44,732 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:55:44,732 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:44,732 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:55:44,734 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:55:44,734 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:44,734 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:55:44,735 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:44,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:44,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:44,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:44,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:44,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:44,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:44,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:44,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:44,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:44,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:44,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:55:44,740 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:55:44,740 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:55:44,740 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:55:44,740 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:55:44,740 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:55:44,740 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:55:44,743 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:44,743 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:44,743 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:44,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:44,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:44,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:44,743 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:44,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:44,744 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:44,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:44,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:44,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:44,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:44,744 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:44,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:44,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:44,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:44,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:44,747 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:44,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:44,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:44,750 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:44,750 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:44,750 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:44,750 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:44,750 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:44,750 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:44,750 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:44,750 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:44,750 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:44,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:44,751 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:44,751 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:44,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:44,752 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:44,752 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:55:44,752 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:55:44,752 DEBUG [Thread-3183] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:55:44,752 DEBUG [Thread-3183] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:55:44,807 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(9158): Flush requested on 0abd14932d4ec4e5fdc6177f2c7cc55c 2023-04-16 00:55:44,811 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 0abd14932d4ec4e5fdc6177f2c7cc55c 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-16 00:55:44,812 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0abd14932d4ec4e5fdc6177f2c7cc55c, store=A 2023-04-16 00:55:44,812 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:55:44,812 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0abd14932d4ec4e5fdc6177f2c7cc55c, store=B 2023-04-16 00:55:44,812 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:55:44,813 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0abd14932d4ec4e5fdc6177f2c7cc55c, store=C 2023-04-16 00:55:44,813 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:55:44,834 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=61 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp/A/e837534d24084370aaa4880531ac951f 2023-04-16 00:55:44,852 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:44,852 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57212 deadline: 1681606604839, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:44,852 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:44,852 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57254 deadline: 1681606604839, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:44,853 DEBUG [Thread-3183] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:55:44,853 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:44,853 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57228 deadline: 1681606604841, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:44,854 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:55:44,856 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:55:44,857 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:55:44,857 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:55:44,857 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:55:44,857 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:44,857 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:55:44,857 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:55:44,857 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:44,858 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:44,858 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:44,858 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57202 deadline: 1681606604851, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:44,862 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:44,862 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:44,862 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:44,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:44,862 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:44,862 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:55:44,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:44,862 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:44,862 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57268 deadline: 1681606604852, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:44,864 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:44,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:55:44,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:44,864 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:55:44,865 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:55:44,865 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:55:44,865 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:44,865 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:55:44,866 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:55:44,866 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:55:44,866 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. 2023-04-16 00:55:44,866 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. started... 2023-04-16 00:55:44,866 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. as already flushing 2023-04-16 00:55:44,867 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. 2023-04-16 00:55:44,867 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:55:44,867 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:55:44,867 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:44,867 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:55:44,867 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:55:44,869 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:44,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:44,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:44,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:44,869 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:44,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:44,870 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:44,870 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:55:44,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:44,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:44,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:44,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:44,871 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:55:44,871 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@302b333[Count = 0] remaining members to acquire global barrier 2023-04-16 00:55:44,871 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:55:44,871 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:44,872 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:44,872 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:44,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:44,872 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:55:44,872 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:55:44,872 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:44,873 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:55:44,873 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:55:44,874 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:44,874 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:55:44,874 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:44,874 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:55:44,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:44,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:44,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:44,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:44,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:44,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:44,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:44,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:44,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:44,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:44,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:55:44,877 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:55:44,877 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:55:44,877 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:55:44,877 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:55:44,877 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:55:44,877 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:55:44,878 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:44,878 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:44,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:44,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:44,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:44,878 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:44,879 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:44,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:44,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:44,879 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:44,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:44,879 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:44,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:44,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:44,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:44,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:44,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:44,880 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:44,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:44,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:44,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:44,884 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:44,884 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:44,884 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:44,884 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:44,884 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:44,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:44,884 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:44,884 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:44,884 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:44,885 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:44,885 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:44,885 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:44,884 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:55:44,885 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:55:44,885 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:44,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:44,885 DEBUG [Thread-3183] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:55:44,885 DEBUG [Thread-3183] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:55:44,954 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:44,954 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57212 deadline: 1681606604953, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:44,954 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:44,955 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57254 deadline: 1681606604954, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:44,955 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:44,955 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57228 deadline: 1681606604954, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:44,965 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:44,965 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57202 deadline: 1681606604960, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:44,967 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:44,968 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57268 deadline: 1681606604964, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:44,985 DEBUG [Thread-3183] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:55:44,986 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:55:44,987 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:55:44,988 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:55:44,988 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:55:44,988 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:55:44,988 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:44,988 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:55:44,988 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:55:44,989 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:44,989 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:44,990 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:44,990 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:44,990 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:44,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:44,991 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:44,991 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:55:44,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:44,991 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:44,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:55:44,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:44,991 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:55:44,992 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:55:44,992 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:55:44,992 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:55:44,992 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:44,992 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:55:44,992 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:55:44,992 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:55:44,992 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. 2023-04-16 00:55:44,992 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. started... 2023-04-16 00:55:44,992 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. as already flushing 2023-04-16 00:55:44,992 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. 2023-04-16 00:55:44,992 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:55:44,992 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:55:44,992 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:44,992 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:55:44,992 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:55:44,994 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:44,994 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:44,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:44,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:44,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:44,994 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:44,994 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:55:44,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:44,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:44,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:44,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:44,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:44,995 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:55:44,995 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:55:44,995 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@9c4a6ea[Count = 0] remaining members to acquire global barrier 2023-04-16 00:55:44,995 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:44,996 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:44,996 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:44,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:44,997 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:55:44,997 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:55:44,997 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:55:44,997 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:44,997 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:55:44,999 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:44,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:44,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:44,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:44,999 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:55:45,000 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:45,000 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:55:45,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:45,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:45,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:45,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:45,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:45,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:45,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:45,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:55:45,001 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:55:45,001 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:55:45,001 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:55:45,001 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:55:45,001 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:55:45,002 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:55:45,003 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:45,003 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:45,003 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:45,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:45,003 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:45,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:45,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:45,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:45,003 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:45,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:45,003 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:45,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:45,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:45,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:45,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:45,005 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:45,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:45,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:45,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:45,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:45,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:45,007 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:45,007 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:45,007 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:45,007 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:45,007 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:45,007 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:45,007 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:55:45,007 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:55:45,007 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:45,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:45,007 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:45,007 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:45,008 DEBUG [Thread-3183] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:55:45,008 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:55:45,008 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:45,008 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:45,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:45,008 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:45,008 DEBUG [Thread-3183] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:55:45,108 DEBUG [Thread-3183] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:55:45,109 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:55:45,110 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:55:45,111 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:55:45,111 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:55:45,111 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:55:45,111 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:45,111 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:55:45,111 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:55:45,112 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:45,112 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:45,113 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:45,113 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:45,113 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:45,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:45,113 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:45,113 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:55:45,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:45,114 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:45,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:55:45,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:45,114 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:55:45,115 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:55:45,115 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:55:45,115 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:55:45,115 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:45,115 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:55:45,115 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:55:45,115 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:55:45,115 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. 2023-04-16 00:55:45,115 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. started... 2023-04-16 00:55:45,116 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. as already flushing 2023-04-16 00:55:45,116 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. 2023-04-16 00:55:45,116 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:55:45,116 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:55:45,116 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:45,116 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:55:45,116 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:55:45,118 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:45,118 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:45,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:45,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:45,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:45,118 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:45,118 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:55:45,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:45,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:45,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:45,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:45,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:45,119 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:55:45,119 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:55:45,119 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7c45bb99[Count = 0] remaining members to acquire global barrier 2023-04-16 00:55:45,120 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:45,121 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:45,121 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:45,121 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:55:45,121 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:45,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:45,121 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:55:45,121 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:55:45,122 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:55:45,123 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:45,123 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:55:45,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:45,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:45,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:45,123 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:45,123 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:55:45,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:45,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:45,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:45,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:45,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:45,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:45,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:45,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:55:45,125 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:55:45,125 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:55:45,125 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:55:45,125 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:55:45,125 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:55:45,125 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:55:45,127 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:45,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:45,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:45,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:45,127 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:45,127 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:45,127 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:45,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:45,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:45,127 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:45,128 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:45,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:45,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:45,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:45,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:45,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:45,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:45,129 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:45,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:45,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:45,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:45,131 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:45,131 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:45,131 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:45,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:45,131 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:45,131 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:45,131 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:45,131 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:45,131 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:55:45,131 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:55:45,131 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:45,131 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:45,131 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:45,131 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:45,132 DEBUG [Thread-3183] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:55:45,132 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:45,132 DEBUG [Thread-3183] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:55:45,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:45,160 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:45,160 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57212 deadline: 1681606605155, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:45,160 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:45,160 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57254 deadline: 1681606605155, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:45,160 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:45,160 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57228 deadline: 1681606605156, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:45,172 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:45,172 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57268 deadline: 1681606605172, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:45,174 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:45,174 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57202 deadline: 1681606605172, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:45,232 DEBUG [Thread-3183] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:55:45,232 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:55:45,234 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:55:45,235 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:55:45,235 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:55:45,235 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:55:45,235 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:45,235 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:55:45,235 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:55:45,236 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:45,236 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:45,237 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:45,237 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:45,237 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:45,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:45,237 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:45,237 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:55:45,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:45,237 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:45,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:55:45,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:45,238 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:55:45,238 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:55:45,238 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:55:45,238 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:55:45,238 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:45,238 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:55:45,238 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:55:45,238 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:55:45,238 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. 2023-04-16 00:55:45,239 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. started... 2023-04-16 00:55:45,239 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. as already flushing 2023-04-16 00:55:45,239 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. 2023-04-16 00:55:45,239 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:55:45,239 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:55:45,239 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:45,239 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:55:45,239 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:55:45,243 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:45,243 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:45,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:45,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:45,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:45,243 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:45,243 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:55:45,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:45,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:45,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:45,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:45,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:45,245 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:55:45,245 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@581e987a[Count = 0] remaining members to acquire global barrier 2023-04-16 00:55:45,245 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:55:45,245 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:45,246 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:45,246 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:45,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:45,246 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:55:45,246 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:55:45,246 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:55:45,247 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:45,247 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:55:45,248 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:55:45,248 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:45,248 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:55:45,248 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:45,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:45,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:45,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:45,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:45,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:45,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:45,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:45,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:45,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:45,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:45,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:55:45,250 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:55:45,250 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:55:45,251 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:55:45,251 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:55:45,251 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:55:45,251 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:55:45,252 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:45,252 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:45,252 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:45,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:45,252 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:45,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:45,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:45,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:45,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:45,253 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:45,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:45,253 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:45,253 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=61 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp/B/784aaa8b6afc4fe58de0a2ebc5d31b38 2023-04-16 00:55:45,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:45,254 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:45,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:45,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:45,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:45,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:45,258 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:45,258 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:45,258 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:45,258 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:45,259 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:45,259 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:45,259 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:45,259 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:45,259 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:45,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:45,259 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:45,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:45,259 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:55:45,259 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:55:45,259 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:45,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:45,259 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:45,259 DEBUG [Thread-3183] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:55:45,259 DEBUG [Thread-3183] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:55:45,260 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:55:45,267 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=61 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp/C/0984f06ceb4740c4a5abfd90bf61adae 2023-04-16 00:55:45,274 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp/A/e837534d24084370aaa4880531ac951f as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/A/e837534d24084370aaa4880531ac951f 2023-04-16 00:55:45,280 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/A/e837534d24084370aaa4880531ac951f, entries=400, sequenceid=61, filesize=23.0 K 2023-04-16 00:55:45,281 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp/B/784aaa8b6afc4fe58de0a2ebc5d31b38 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/B/784aaa8b6afc4fe58de0a2ebc5d31b38 2023-04-16 00:55:45,284 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/B/784aaa8b6afc4fe58de0a2ebc5d31b38, entries=150, sequenceid=61, filesize=11.6 K 2023-04-16 00:55:45,284 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp/C/0984f06ceb4740c4a5abfd90bf61adae as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/C/0984f06ceb4740c4a5abfd90bf61adae 2023-04-16 00:55:45,287 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/C/0984f06ceb4740c4a5abfd90bf61adae, entries=150, sequenceid=61, filesize=11.6 K 2023-04-16 00:55:45,287 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=93.93 KB/96180 for 0abd14932d4ec4e5fdc6177f2c7cc55c in 476ms, sequenceid=61, compaction requested=true 2023-04-16 00:55:45,287 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 0abd14932d4ec4e5fdc6177f2c7cc55c: 2023-04-16 00:55:45,288 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 00:55:45,288 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-16 00:55:45,288 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 00:55:45,288 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-16 00:55:45,288 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 00:55:45,290 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 61588 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-16 00:55:45,290 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47544 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-16 00:55:45,290 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HStore(1912): 0abd14932d4ec4e5fdc6177f2c7cc55c/A is initiating minor compaction (all files) 2023-04-16 00:55:45,290 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HStore(1912): 0abd14932d4ec4e5fdc6177f2c7cc55c/B is initiating minor compaction (all files) 2023-04-16 00:55:45,290 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 0abd14932d4ec4e5fdc6177f2c7cc55c/A in TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. 2023-04-16 00:55:45,290 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 0abd14932d4ec4e5fdc6177f2c7cc55c/B in TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. 2023-04-16 00:55:45,290 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/A/71418beef49441f398ffb9fdee497861, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/A/f7a21da1edc6453a87700f88d4768b2c, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/A/a536085eda52409db66262bfc6d27349, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/A/e837534d24084370aaa4880531ac951f] into tmpdir=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp, totalSize=60.1 K 2023-04-16 00:55:45,291 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/B/f4aad6d467e94184902a4a8943a66c2d, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/B/2446ae0deb3c4113b943f0f34e7e4b9a, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/B/87dac584dc4248e28d98db785d1b2d4e, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/B/784aaa8b6afc4fe58de0a2ebc5d31b38] into tmpdir=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp, totalSize=46.4 K 2023-04-16 00:55:45,291 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.Compactor(207): Compacting 71418beef49441f398ffb9fdee497861, keycount=200, bloomtype=ROW, size=13.9 K, encoding=NONE, compression=NONE, seqNum=14, earliestPutTs=1681606542951 2023-04-16 00:55:45,291 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting f4aad6d467e94184902a4a8943a66c2d, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=14, earliestPutTs=1681606542951 2023-04-16 00:55:45,291 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.Compactor(207): Compacting f7a21da1edc6453a87700f88d4768b2c, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=34, earliestPutTs=1681606543022 2023-04-16 00:55:45,291 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting 2446ae0deb3c4113b943f0f34e7e4b9a, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=34, earliestPutTs=1681606543022 2023-04-16 00:55:45,292 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.Compactor(207): Compacting a536085eda52409db66262bfc6d27349, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=46, earliestPutTs=1681606543650 2023-04-16 00:55:45,292 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting 87dac584dc4248e28d98db785d1b2d4e, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=46, earliestPutTs=1681606543650 2023-04-16 00:55:45,292 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.Compactor(207): Compacting e837534d24084370aaa4880531ac951f, keycount=400, bloomtype=ROW, size=23.0 K, encoding=NONE, compression=NONE, seqNum=61, earliestPutTs=1681606544800 2023-04-16 00:55:45,292 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting 784aaa8b6afc4fe58de0a2ebc5d31b38, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=61, earliestPutTs=1681606544810 2023-04-16 00:55:45,300 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] throttle.PressureAwareThroughputController(145): 0abd14932d4ec4e5fdc6177f2c7cc55c#A#compaction#417 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-16 00:55:45,301 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] throttle.PressureAwareThroughputController(145): 0abd14932d4ec4e5fdc6177f2c7cc55c#B#compaction#418 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-16 00:55:45,359 DEBUG [Thread-3183] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:55:45,360 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:55:45,361 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:55:45,362 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:55:45,362 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:55:45,362 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:55:45,362 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:45,362 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:55:45,362 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:55:45,363 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:45,363 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:45,364 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:45,364 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:45,364 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:45,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:45,364 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:45,364 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:55:45,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:45,365 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:45,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:55:45,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:45,365 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:55:45,365 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:55:45,365 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:55:45,365 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:55:45,365 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:45,365 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:55:45,365 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:55:45,365 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:55:45,366 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. 2023-04-16 00:55:45,366 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. started... 2023-04-16 00:55:45,366 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 0abd14932d4ec4e5fdc6177f2c7cc55c 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-04-16 00:55:45,366 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0abd14932d4ec4e5fdc6177f2c7cc55c, store=A 2023-04-16 00:55:45,366 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:55:45,366 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0abd14932d4ec4e5fdc6177f2c7cc55c, store=B 2023-04-16 00:55:45,366 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:55:45,366 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0abd14932d4ec4e5fdc6177f2c7cc55c, store=C 2023-04-16 00:55:45,366 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:55:45,373 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=78 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp/A/a7aea499e2c74e8ea2c5d13b76636911 2023-04-16 00:55:45,470 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(9158): Flush requested on 0abd14932d4ec4e5fdc6177f2c7cc55c 2023-04-16 00:55:45,470 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. as already flushing 2023-04-16 00:55:45,496 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:45,496 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57202 deadline: 1681606605483, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:45,503 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:45,503 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57228 deadline: 1681606605491, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:45,507 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:45,507 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57212 deadline: 1681606605494, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:45,507 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:45,508 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57254 deadline: 1681606605495, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:45,509 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:45,509 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57268 deadline: 1681606605496, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:45,600 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:45,600 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57202 deadline: 1681606605597, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:45,610 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:45,610 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57228 deadline: 1681606605604, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:45,611 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:45,611 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57212 deadline: 1681606605608, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:45,611 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:45,611 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57254 deadline: 1681606605608, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:45,621 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:45,621 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57268 deadline: 1681606605610, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:45,653 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-16 00:55:45,722 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp/A/cb45045553bf4063b4c5318114b406e7 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/A/cb45045553bf4063b4c5318114b406e7 2023-04-16 00:55:45,723 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp/B/f0973d70203e443aa76f898d60d85c6b as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/B/f0973d70203e443aa76f898d60d85c6b 2023-04-16 00:55:45,726 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 0abd14932d4ec4e5fdc6177f2c7cc55c/A of 0abd14932d4ec4e5fdc6177f2c7cc55c into cb45045553bf4063b4c5318114b406e7(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 00:55:45,727 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0abd14932d4ec4e5fdc6177f2c7cc55c: 2023-04-16 00:55:45,727 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c., storeName=0abd14932d4ec4e5fdc6177f2c7cc55c/A, priority=12, startTime=1681606545288; duration=0sec 2023-04-16 00:55:45,727 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 00:55:45,727 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-16 00:55:45,728 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 0abd14932d4ec4e5fdc6177f2c7cc55c/B of 0abd14932d4ec4e5fdc6177f2c7cc55c into f0973d70203e443aa76f898d60d85c6b(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 00:55:45,728 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0abd14932d4ec4e5fdc6177f2c7cc55c: 2023-04-16 00:55:45,728 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47544 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-16 00:55:45,728 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c., storeName=0abd14932d4ec4e5fdc6177f2c7cc55c/B, priority=12, startTime=1681606545288; duration=0sec 2023-04-16 00:55:45,728 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HStore(1912): 0abd14932d4ec4e5fdc6177f2c7cc55c/C is initiating minor compaction (all files) 2023-04-16 00:55:45,729 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 00:55:45,729 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 0abd14932d4ec4e5fdc6177f2c7cc55c/C in TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. 2023-04-16 00:55:45,729 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/C/a73b04d521c54866abefb4e0094431e3, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/C/330a03ff6a924906b546600756cc25ea, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/C/eae239871d0845019a296960b5d4fd40, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/C/0984f06ceb4740c4a5abfd90bf61adae] into tmpdir=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp, totalSize=46.4 K 2023-04-16 00:55:45,729 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.Compactor(207): Compacting a73b04d521c54866abefb4e0094431e3, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=14, earliestPutTs=1681606542951 2023-04-16 00:55:45,729 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.Compactor(207): Compacting 330a03ff6a924906b546600756cc25ea, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=34, earliestPutTs=1681606543022 2023-04-16 00:55:45,730 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.Compactor(207): Compacting eae239871d0845019a296960b5d4fd40, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=46, earliestPutTs=1681606543650 2023-04-16 00:55:45,730 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.Compactor(207): Compacting 0984f06ceb4740c4a5abfd90bf61adae, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=61, earliestPutTs=1681606544810 2023-04-16 00:55:45,737 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] throttle.PressureAwareThroughputController(145): 0abd14932d4ec4e5fdc6177f2c7cc55c#C#compaction#421 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-16 00:55:45,746 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp/C/a3847ebac68d4ac48054ed2e5dfeec04 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/C/a3847ebac68d4ac48054ed2e5dfeec04 2023-04-16 00:55:45,750 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 0abd14932d4ec4e5fdc6177f2c7cc55c/C of 0abd14932d4ec4e5fdc6177f2c7cc55c into a3847ebac68d4ac48054ed2e5dfeec04(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 00:55:45,750 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0abd14932d4ec4e5fdc6177f2c7cc55c: 2023-04-16 00:55:45,750 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c., storeName=0abd14932d4ec4e5fdc6177f2c7cc55c/C, priority=12, startTime=1681606545288; duration=0sec 2023-04-16 00:55:45,750 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 00:55:45,787 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=78 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp/B/134f3fae05af47148ac0f7d284854e20 2023-04-16 00:55:45,804 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:45,805 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57202 deadline: 1681606605801, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:45,814 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:45,815 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57228 deadline: 1681606605811, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:45,815 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:45,815 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57212 deadline: 1681606605812, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:45,815 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:45,815 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57254 deadline: 1681606605812, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:45,825 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:45,825 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57268 deadline: 1681606605822, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:46,106 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:46,106 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57202 deadline: 1681606606106, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:46,120 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:46,120 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57228 deadline: 1681606606115, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:46,120 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:46,120 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57254 deadline: 1681606606117, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:46,120 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:46,120 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57212 deadline: 1681606606118, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:46,131 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:46,131 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57268 deadline: 1681606606127, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:46,197 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=78 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp/C/0adf4d55f31843248cc219e790d480ce 2023-04-16 00:55:46,201 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp/A/a7aea499e2c74e8ea2c5d13b76636911 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/A/a7aea499e2c74e8ea2c5d13b76636911 2023-04-16 00:55:46,205 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/A/a7aea499e2c74e8ea2c5d13b76636911, entries=150, sequenceid=78, filesize=11.6 K 2023-04-16 00:55:46,206 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp/B/134f3fae05af47148ac0f7d284854e20 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/B/134f3fae05af47148ac0f7d284854e20 2023-04-16 00:55:46,209 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/B/134f3fae05af47148ac0f7d284854e20, entries=150, sequenceid=78, filesize=11.6 K 2023-04-16 00:55:46,210 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp/C/0adf4d55f31843248cc219e790d480ce as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/C/0adf4d55f31843248cc219e790d480ce 2023-04-16 00:55:46,213 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/C/0adf4d55f31843248cc219e790d480ce, entries=150, sequenceid=78, filesize=11.6 K 2023-04-16 00:55:46,214 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=80.51 KB/82440 for 0abd14932d4ec4e5fdc6177f2c7cc55c in 848ms, sequenceid=78, compaction requested=false 2023-04-16 00:55:46,214 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 0abd14932d4ec4e5fdc6177f2c7cc55c: 2023-04-16 00:55:46,214 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. 2023-04-16 00:55:46,214 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:55:46,214 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:55:46,214 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:46,214 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:55:46,214 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:55:46,215 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:46,215 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:46,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:46,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:46,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:46,216 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:46,216 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:55:46,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:46,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:46,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:46,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:46,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:46,217 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:55:46,218 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4e132361[Count = 0] remaining members to acquire global barrier 2023-04-16 00:55:46,218 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:55:46,218 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:46,219 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:46,219 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:46,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:46,219 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:55:46,219 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:55:46,219 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:55:46,219 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:46,219 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:55:46,223 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:55:46,223 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:46,223 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:55:46,223 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:46,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:46,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:46,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:46,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:46,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:46,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:46,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:46,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:46,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:46,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:46,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:55:46,226 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:55:46,226 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:55:46,226 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:55:46,226 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:55:46,226 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:55:46,226 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:55:46,227 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:46,227 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:46,227 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:46,227 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:46,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:46,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:46,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:46,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:46,228 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:46,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:46,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:46,228 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:46,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:46,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:46,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:46,230 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:46,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:46,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:46,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:46,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:46,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:46,232 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:46,232 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:46,232 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:46,232 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:46,232 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:46,232 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:46,232 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:55:46,232 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:46,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:46,232 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:46,233 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:46,232 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:55:46,232 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:46,233 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:55:46,233 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:46,233 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:46,233 DEBUG [Thread-3183] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:55:46,233 DEBUG [Thread-3183] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:55:46,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:46,333 DEBUG [Thread-3183] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:55:46,334 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:55:46,335 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:55:46,336 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:55:46,336 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:55:46,336 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:55:46,336 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:46,336 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:55:46,336 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:55:46,337 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:46,337 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:46,338 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:46,338 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:46,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:46,338 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:46,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:46,338 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:46,338 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:55:46,338 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:46,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:55:46,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:46,339 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:55:46,339 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:55:46,339 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:55:46,339 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:55:46,339 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:46,339 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:55:46,339 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:55:46,339 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:55:46,340 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. 2023-04-16 00:55:46,340 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. started... 2023-04-16 00:55:46,340 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 0abd14932d4ec4e5fdc6177f2c7cc55c 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-04-16 00:55:46,340 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0abd14932d4ec4e5fdc6177f2c7cc55c, store=A 2023-04-16 00:55:46,340 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:55:46,340 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0abd14932d4ec4e5fdc6177f2c7cc55c, store=B 2023-04-16 00:55:46,340 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:55:46,340 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0abd14932d4ec4e5fdc6177f2c7cc55c, store=C 2023-04-16 00:55:46,340 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:55:46,347 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=96 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp/A/cdc912a5a9c948e7b803f03edf8cfa6e 2023-04-16 00:55:46,359 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=96 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp/B/4be52027907649ae8dc19e46d5332468 2023-04-16 00:55:46,368 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=96 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp/C/d72bce8c83a249a391d08c9d6c3a5a16 2023-04-16 00:55:46,371 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp/A/cdc912a5a9c948e7b803f03edf8cfa6e as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/A/cdc912a5a9c948e7b803f03edf8cfa6e 2023-04-16 00:55:46,374 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/A/cdc912a5a9c948e7b803f03edf8cfa6e, entries=150, sequenceid=96, filesize=11.6 K 2023-04-16 00:55:46,375 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp/B/4be52027907649ae8dc19e46d5332468 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/B/4be52027907649ae8dc19e46d5332468 2023-04-16 00:55:46,378 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/B/4be52027907649ae8dc19e46d5332468, entries=150, sequenceid=96, filesize=11.6 K 2023-04-16 00:55:46,379 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp/C/d72bce8c83a249a391d08c9d6c3a5a16 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/C/d72bce8c83a249a391d08c9d6c3a5a16 2023-04-16 00:55:46,385 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/C/d72bce8c83a249a391d08c9d6c3a5a16, entries=150, sequenceid=96, filesize=11.6 K 2023-04-16 00:55:46,385 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=0 B/0 for 0abd14932d4ec4e5fdc6177f2c7cc55c in 45ms, sequenceid=96, compaction requested=true 2023-04-16 00:55:46,386 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 0abd14932d4ec4e5fdc6177f2c7cc55c: 2023-04-16 00:55:46,386 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. 2023-04-16 00:55:46,386 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:55:46,386 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:55:46,386 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:46,386 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:55:46,386 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:55:46,392 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:46,392 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:46,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:46,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:46,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:46,392 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:46,392 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:55:46,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:46,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:46,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:46,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:46,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:46,394 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:55:46,394 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@62cc3fbf[Count = 0] remaining members to acquire global barrier 2023-04-16 00:55:46,394 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:55:46,394 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:46,395 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:46,395 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:46,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:46,395 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:55:46,395 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:55:46,395 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:46,395 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:55:46,395 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:55:46,397 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:46,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:46,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:46,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:46,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:46,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:46,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:46,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:46,398 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:55:46,398 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:46,398 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:55:46,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:46,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:46,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:46,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:55:46,399 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:55:46,399 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:55:46,399 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:55:46,399 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:55:46,399 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:55:46,399 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:55:46,400 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:46,400 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:46,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:46,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:46,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:46,400 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:46,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:46,400 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:46,401 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:46,401 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:46,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:46,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:46,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:46,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:46,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:46,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:46,402 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:46,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:46,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:46,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:46,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:46,408 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:46,408 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:46,409 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:46,409 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:46,409 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:55:46,409 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:55:46,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:46,409 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:46,409 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:46,409 DEBUG [Thread-3183] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:55:46,409 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:46,409 DEBUG [Thread-3183] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:55:46,409 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:46,409 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:46,409 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:46,409 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:46,409 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:46,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:46,410 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:55:46,509 DEBUG [Thread-3183] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:55:46,510 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:55:46,511 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:55:46,512 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:55:46,512 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:55:46,512 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:55:46,512 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:46,512 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:55:46,512 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:55:46,513 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:46,513 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:46,514 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:46,514 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:46,514 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:46,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:46,514 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:46,514 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:55:46,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:46,515 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:46,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:55:46,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:46,515 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:55:46,515 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:55:46,515 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:55:46,515 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:55:46,515 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:46,515 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:55:46,515 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:55:46,515 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:55:46,515 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. 2023-04-16 00:55:46,516 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. started... 2023-04-16 00:55:46,516 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 0abd14932d4ec4e5fdc6177f2c7cc55c: 2023-04-16 00:55:46,516 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. 2023-04-16 00:55:46,516 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:55:46,516 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:55:46,516 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:46,516 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:55:46,516 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:55:46,517 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:46,517 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:46,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:46,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:46,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:46,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:46,518 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:46,518 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:55:46,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:46,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:46,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:46,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:46,520 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:55:46,520 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3384c90c[Count = 0] remaining members to acquire global barrier 2023-04-16 00:55:46,520 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:55:46,520 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:46,521 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:46,521 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:46,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:46,521 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:55:46,521 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:55:46,521 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:55:46,521 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:46,521 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:55:46,526 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:46,526 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:55:46,526 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:46,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:46,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:46,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:46,526 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:55:46,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:46,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:46,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:46,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:46,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:46,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:46,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:46,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:55:46,529 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:55:46,529 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:55:46,529 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:55:46,529 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:55:46,529 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:55:46,529 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:55:46,531 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:46,531 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:46,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:46,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:46,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:46,531 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:46,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:46,531 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:46,532 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:46,532 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:46,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:46,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:46,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:46,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:46,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:46,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:46,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:46,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:46,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:46,534 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:46,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:46,536 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:46,536 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:46,536 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:46,536 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:46,536 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:55:46,536 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:55:46,536 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:46,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:46,536 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:46,536 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:46,536 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:46,536 DEBUG [Thread-3183] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:55:46,536 DEBUG [Thread-3183] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:55:46,536 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:46,536 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:46,536 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:46,537 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:46,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:46,628 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(9158): Flush requested on 0abd14932d4ec4e5fdc6177f2c7cc55c 2023-04-16 00:55:46,628 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 0abd14932d4ec4e5fdc6177f2c7cc55c 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-16 00:55:46,629 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0abd14932d4ec4e5fdc6177f2c7cc55c, store=A 2023-04-16 00:55:46,629 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:55:46,629 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0abd14932d4ec4e5fdc6177f2c7cc55c, store=B 2023-04-16 00:55:46,630 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:55:46,630 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0abd14932d4ec4e5fdc6177f2c7cc55c, store=C 2023-04-16 00:55:46,630 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:55:46,636 DEBUG [Thread-3183] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:55:46,637 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=108 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp/A/8481f81a8bbc45fda145aeed58997fd7 2023-04-16 00:55:46,637 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:55:46,639 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:55:46,640 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:55:46,640 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:55:46,640 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:55:46,640 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:46,640 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:55:46,640 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:55:46,641 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:46,641 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:46,642 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:46,642 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:46,642 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:46,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:46,642 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:46,642 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:55:46,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:46,642 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:46,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:55:46,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:46,643 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:55:46,643 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:55:46,643 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:55:46,643 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:55:46,643 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:46,643 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:55:46,643 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:55:46,643 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:55:46,643 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. 2023-04-16 00:55:46,644 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. started... 2023-04-16 00:55:46,644 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. as already flushing 2023-04-16 00:55:46,644 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. 2023-04-16 00:55:46,644 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:55:46,644 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:55:46,644 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:46,644 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:55:46,644 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:55:46,646 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:46,646 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:46,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:46,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:46,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:46,646 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:46,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:46,646 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:55:46,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:46,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:46,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:46,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:46,648 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:55:46,648 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1ff55a4e[Count = 0] remaining members to acquire global barrier 2023-04-16 00:55:46,648 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:55:46,648 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:46,650 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:46,650 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:46,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:46,650 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:55:46,650 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:55:46,650 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:55:46,650 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:46,650 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:55:46,651 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:55:46,651 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:46,651 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:46,651 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:55:46,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:46,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:46,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:46,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:46,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:46,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:46,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:46,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:46,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:46,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:46,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:55:46,655 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:55:46,655 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:55:46,655 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:55:46,655 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:55:46,655 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:55:46,655 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:55:46,656 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:46,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:46,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:46,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:46,656 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:46,657 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:46,657 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:46,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:46,657 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:46,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:46,657 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:46,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:46,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:46,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:46,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:46,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:46,658 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:46,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:46,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:46,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:46,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:46,661 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:46,661 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:46,661 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:46,661 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:46,661 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:46,661 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:46,661 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:46,661 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:46,661 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:46,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:46,661 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:46,661 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:46,661 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:55:46,661 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:55:46,662 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:46,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:46,662 DEBUG [Thread-3183] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:55:46,662 DEBUG [Thread-3183] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:55:46,662 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:55:46,665 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:46,665 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57228 deadline: 1681606606657, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:46,665 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1681606546655, max timestamp=1681606546663Num uniques 859; 2023-04-16 00:55:46,666 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-16 00:55:46,666 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1681606546655, max timestamp=1681606546663Num uniques 859; 2023-04-16 00:55:46,666 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1681606546655, max timestamp=1681606546663Num uniques 859; 2023-04-16 00:55:46,666 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-16 00:55:46,666 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-16 00:55:46,700 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:46,701 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57202 deadline: 1681606606689, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:46,701 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:46,701 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57268 deadline: 1681606606692, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:46,701 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:46,701 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57254 deadline: 1681606606696, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:46,708 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:46,708 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57212 deadline: 1681606606701, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:46,762 DEBUG [Thread-3183] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:55:46,762 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:55:46,764 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:55:46,765 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:55:46,765 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:55:46,765 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:55:46,765 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:46,766 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:55:46,766 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:55:46,766 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:46,766 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:46,767 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:46,767 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:46,767 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:46,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:46,768 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:46,768 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:55:46,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:46,768 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:46,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:55:46,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:46,768 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:55:46,769 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:55:46,769 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:55:46,769 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:55:46,769 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:46,769 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:55:46,770 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:55:46,770 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:55:46,770 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. 2023-04-16 00:55:46,770 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. started... 2023-04-16 00:55:46,770 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. as already flushing 2023-04-16 00:55:46,770 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. 2023-04-16 00:55:46,770 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:55:46,770 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:55:46,770 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:46,770 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:55:46,770 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:55:46,771 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:46,772 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57228 deadline: 1681606606767, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:46,772 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:46,772 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:46,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:46,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:46,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:46,772 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:46,772 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:55:46,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:46,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:46,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:46,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:46,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:46,774 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:55:46,774 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@48aa6afa[Count = 0] remaining members to acquire global barrier 2023-04-16 00:55:46,774 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:55:46,774 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:46,775 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:46,775 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:46,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:46,775 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:55:46,775 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:55:46,775 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:55:46,775 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:46,775 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:55:46,779 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:46,779 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:55:46,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:46,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:46,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:46,779 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:46,779 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:55:46,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:46,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:46,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:46,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:46,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:46,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:46,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:46,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:55:46,782 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:55:46,782 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:55:46,782 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:55:46,783 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:55:46,783 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:55:46,783 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:55:46,784 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:46,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:46,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:46,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:46,785 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:46,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:46,786 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:46,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:46,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:46,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:46,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:46,788 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:46,788 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:46,788 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:46,788 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:46,788 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:55:46,788 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:55:46,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:46,788 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:46,789 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:46,789 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:46,789 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:46,789 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:46,789 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:46,789 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:46,789 DEBUG [Thread-3183] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:55:46,789 DEBUG [Thread-3183] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:55:46,789 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:46,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:46,789 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:55:46,789 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:55:46,789 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:46,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:46,790 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:46,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:46,805 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:46,806 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57202 deadline: 1681606606802, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:46,806 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:46,806 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57268 deadline: 1681606606802, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:46,806 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:46,806 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57254 deadline: 1681606606802, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:46,819 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:46,819 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57212 deadline: 1681606606809, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:46,889 DEBUG [Thread-3183] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:55:46,889 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:55:46,891 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:55:46,892 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:55:46,892 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:55:46,892 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:55:46,892 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:46,892 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:55:46,892 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:55:46,893 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:46,893 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:46,894 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:46,894 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:46,894 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:46,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:46,894 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:46,894 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:55:46,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:46,894 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:46,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:55:46,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:46,895 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:55:46,895 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:55:46,895 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:55:46,895 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:55:46,895 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:46,895 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:55:46,896 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:55:46,896 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:55:46,896 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. 2023-04-16 00:55:46,896 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. started... 2023-04-16 00:55:46,896 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. as already flushing 2023-04-16 00:55:46,896 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. 2023-04-16 00:55:46,896 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:55:46,897 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:55:46,897 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:46,897 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:55:46,897 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:55:46,904 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:46,904 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:46,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:46,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:46,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:46,905 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:46,905 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:55:46,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:46,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:46,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:46,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:46,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:46,906 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:55:46,906 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4b188597[Count = 0] remaining members to acquire global barrier 2023-04-16 00:55:46,906 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:55:46,906 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:46,907 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:46,908 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:46,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:46,908 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:55:46,908 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:55:46,908 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:55:46,908 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:46,908 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:55:46,909 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:55:46,909 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:46,909 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:46,909 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:55:46,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:46,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:46,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:46,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:46,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:46,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:46,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:46,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:46,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:46,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:46,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:55:46,912 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:55:46,912 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:55:46,912 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:55:46,912 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:55:46,912 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:55:46,912 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:55:46,922 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:46,922 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:46,922 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:46,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:46,922 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:46,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:46,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:46,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:46,922 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:46,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:46,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:46,922 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:46,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:46,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:46,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:46,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:46,924 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:46,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:46,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:46,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:46,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:46,931 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:46,931 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:46,931 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:46,931 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:46,931 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:46,931 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:55:46,931 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:55:46,931 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:46,931 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:46,931 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:46,931 DEBUG [Thread-3183] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:55:46,931 DEBUG [Thread-3183] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:55:46,931 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:46,931 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:46,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:46,931 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:46,932 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:46,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:46,979 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:46,979 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57228 deadline: 1681606606973, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:47,010 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:47,011 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57202 deadline: 1681606607007, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:47,011 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:47,011 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57268 deadline: 1681606607007, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:47,019 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:47,019 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57254 deadline: 1681606607008, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:47,031 DEBUG [Thread-3183] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:55:47,032 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:55:47,033 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:47,033 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57212 deadline: 1681606607021, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:47,033 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:55:47,034 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:55:47,034 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:55:47,035 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:55:47,035 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:47,035 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:55:47,035 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:55:47,035 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:47,035 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:47,036 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:47,036 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:47,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:47,036 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:47,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:47,037 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:47,037 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:55:47,037 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:47,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:55:47,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:47,037 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:55:47,038 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:55:47,038 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:55:47,038 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:55:47,038 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:47,038 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:55:47,038 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:55:47,038 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:55:47,038 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. 2023-04-16 00:55:47,038 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. started... 2023-04-16 00:55:47,038 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. as already flushing 2023-04-16 00:55:47,038 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. 2023-04-16 00:55:47,038 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:55:47,038 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:55:47,039 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:47,039 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:55:47,039 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:55:47,040 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:47,040 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:47,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:47,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:47,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:47,040 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:47,041 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:55:47,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:47,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:47,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:47,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:47,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:47,042 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:55:47,042 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5ec1e951[Count = 0] remaining members to acquire global barrier 2023-04-16 00:55:47,042 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:55:47,042 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:47,044 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:47,044 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:47,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:47,044 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:55:47,044 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:55:47,044 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:55:47,044 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:47,044 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:55:47,045 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:55:47,045 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:47,045 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:47,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:47,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:47,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:47,045 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:55:47,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:47,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:47,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:47,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:47,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:47,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:47,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:47,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:55:47,048 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:55:47,048 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:55:47,048 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:55:47,048 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:55:47,048 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:55:47,048 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:55:47,049 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:47,049 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:47,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:47,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:47,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:47,049 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:47,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:47,049 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:47,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:47,050 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:47,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:47,050 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:47,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:47,052 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=108 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp/B/7f05a6d097444efdab1b362d137e0f01 2023-04-16 00:55:47,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:47,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:47,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:47,053 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:47,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:47,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:47,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:47,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:47,057 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:47,057 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:47,057 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:47,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:47,057 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:47,057 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:55:47,057 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:55:47,058 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:47,058 DEBUG [Thread-3183] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:55:47,058 DEBUG [Thread-3183] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:55:47,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:47,059 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:47,059 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:47,059 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:47,059 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:47,060 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:47,060 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:47,060 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:47,060 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:55:47,063 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=108 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp/C/b787863c7ff84cfdb20b1010584fd1ac 2023-04-16 00:55:47,067 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp/A/8481f81a8bbc45fda145aeed58997fd7 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/A/8481f81a8bbc45fda145aeed58997fd7 2023-04-16 00:55:47,072 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/A/8481f81a8bbc45fda145aeed58997fd7, entries=250, sequenceid=108, filesize=16.2 K 2023-04-16 00:55:47,072 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp/B/7f05a6d097444efdab1b362d137e0f01 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/B/7f05a6d097444efdab1b362d137e0f01 2023-04-16 00:55:47,076 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/B/7f05a6d097444efdab1b362d137e0f01, entries=150, sequenceid=108, filesize=11.6 K 2023-04-16 00:55:47,077 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp/C/b787863c7ff84cfdb20b1010584fd1ac as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/C/b787863c7ff84cfdb20b1010584fd1ac 2023-04-16 00:55:47,083 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/C/b787863c7ff84cfdb20b1010584fd1ac, entries=150, sequenceid=108, filesize=11.6 K 2023-04-16 00:55:47,083 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 0abd14932d4ec4e5fdc6177f2c7cc55c in 455ms, sequenceid=108, compaction requested=true 2023-04-16 00:55:47,083 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 0abd14932d4ec4e5fdc6177f2c7cc55c: 2023-04-16 00:55:47,084 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 00:55:47,084 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-16 00:55:47,084 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-16 00:55:47,084 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 00:55:47,084 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 00:55:47,085 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 52362 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-16 00:55:47,085 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HStore(1912): 0abd14932d4ec4e5fdc6177f2c7cc55c/A is initiating minor compaction (all files) 2023-04-16 00:55:47,085 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 0abd14932d4ec4e5fdc6177f2c7cc55c/A in TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. 2023-04-16 00:55:47,085 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/A/cb45045553bf4063b4c5318114b406e7, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/A/a7aea499e2c74e8ea2c5d13b76636911, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/A/cdc912a5a9c948e7b803f03edf8cfa6e, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/A/8481f81a8bbc45fda145aeed58997fd7] into tmpdir=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp, totalSize=51.1 K 2023-04-16 00:55:47,086 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.Compactor(207): Compacting cb45045553bf4063b4c5318114b406e7, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=61, earliestPutTs=1681606544810 2023-04-16 00:55:47,086 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47682 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-16 00:55:47,086 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HStore(1912): 0abd14932d4ec4e5fdc6177f2c7cc55c/B is initiating minor compaction (all files) 2023-04-16 00:55:47,086 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 0abd14932d4ec4e5fdc6177f2c7cc55c/B in TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. 2023-04-16 00:55:47,086 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/B/f0973d70203e443aa76f898d60d85c6b, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/B/134f3fae05af47148ac0f7d284854e20, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/B/4be52027907649ae8dc19e46d5332468, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/B/7f05a6d097444efdab1b362d137e0f01] into tmpdir=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp, totalSize=46.6 K 2023-04-16 00:55:47,086 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.Compactor(207): Compacting a7aea499e2c74e8ea2c5d13b76636911, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=78, earliestPutTs=1681606544832 2023-04-16 00:55:47,086 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting f0973d70203e443aa76f898d60d85c6b, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=61, earliestPutTs=1681606544810 2023-04-16 00:55:47,087 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.Compactor(207): Compacting cdc912a5a9c948e7b803f03edf8cfa6e, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=96, earliestPutTs=1681606545481 2023-04-16 00:55:47,087 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting 134f3fae05af47148ac0f7d284854e20, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=78, earliestPutTs=1681606544832 2023-04-16 00:55:47,087 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.Compactor(207): Compacting 8481f81a8bbc45fda145aeed58997fd7, keycount=250, bloomtype=ROW, size=16.2 K, encoding=NONE, compression=NONE, seqNum=108, earliestPutTs=1681606546611 2023-04-16 00:55:47,087 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting 4be52027907649ae8dc19e46d5332468, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=96, earliestPutTs=1681606545481 2023-04-16 00:55:47,088 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting 7f05a6d097444efdab1b362d137e0f01, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=108, earliestPutTs=1681606546611 2023-04-16 00:55:47,095 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] throttle.PressureAwareThroughputController(145): 0abd14932d4ec4e5fdc6177f2c7cc55c#B#compaction#429 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-16 00:55:47,102 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] throttle.PressureAwareThroughputController(145): 0abd14932d4ec4e5fdc6177f2c7cc55c#A#compaction#430 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-16 00:55:47,112 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp/B/51638d2847714f59a9001c13cc9b1ddf as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/B/51638d2847714f59a9001c13cc9b1ddf 2023-04-16 00:55:47,119 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 0abd14932d4ec4e5fdc6177f2c7cc55c/B of 0abd14932d4ec4e5fdc6177f2c7cc55c into 51638d2847714f59a9001c13cc9b1ddf(size=11.9 K), total size for store is 11.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 00:55:47,119 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0abd14932d4ec4e5fdc6177f2c7cc55c: 2023-04-16 00:55:47,119 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c., storeName=0abd14932d4ec4e5fdc6177f2c7cc55c/B, priority=12, startTime=1681606547084; duration=0sec 2023-04-16 00:55:47,119 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 00:55:47,119 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-16 00:55:47,120 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp/A/7296aeea67a740a3baa8203cdefc1ba8 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/A/7296aeea67a740a3baa8203cdefc1ba8 2023-04-16 00:55:47,121 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47682 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-16 00:55:47,121 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HStore(1912): 0abd14932d4ec4e5fdc6177f2c7cc55c/C is initiating minor compaction (all files) 2023-04-16 00:55:47,121 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 0abd14932d4ec4e5fdc6177f2c7cc55c/C in TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. 2023-04-16 00:55:47,121 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/C/a3847ebac68d4ac48054ed2e5dfeec04, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/C/0adf4d55f31843248cc219e790d480ce, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/C/d72bce8c83a249a391d08c9d6c3a5a16, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/C/b787863c7ff84cfdb20b1010584fd1ac] into tmpdir=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp, totalSize=46.6 K 2023-04-16 00:55:47,121 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting a3847ebac68d4ac48054ed2e5dfeec04, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=61, earliestPutTs=1681606544810 2023-04-16 00:55:47,121 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting 0adf4d55f31843248cc219e790d480ce, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=78, earliestPutTs=1681606544832 2023-04-16 00:55:47,122 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting d72bce8c83a249a391d08c9d6c3a5a16, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=96, earliestPutTs=1681606545481 2023-04-16 00:55:47,122 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting b787863c7ff84cfdb20b1010584fd1ac, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=108, earliestPutTs=1681606546611 2023-04-16 00:55:47,129 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 0abd14932d4ec4e5fdc6177f2c7cc55c/A of 0abd14932d4ec4e5fdc6177f2c7cc55c into 7296aeea67a740a3baa8203cdefc1ba8(size=11.9 K), total size for store is 11.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 00:55:47,129 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0abd14932d4ec4e5fdc6177f2c7cc55c: 2023-04-16 00:55:47,129 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c., storeName=0abd14932d4ec4e5fdc6177f2c7cc55c/A, priority=12, startTime=1681606547084; duration=0sec 2023-04-16 00:55:47,129 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 00:55:47,145 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] throttle.PressureAwareThroughputController(145): 0abd14932d4ec4e5fdc6177f2c7cc55c#C#compaction#431 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-16 00:55:47,158 DEBUG [Thread-3183] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:55:47,159 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp/C/1717d4eff3cf4343bf0e1fd244864b28 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/C/1717d4eff3cf4343bf0e1fd244864b28 2023-04-16 00:55:47,159 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:55:47,161 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:55:47,162 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:55:47,162 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:55:47,162 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:55:47,162 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:47,162 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:55:47,162 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:55:47,162 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:47,162 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:47,164 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 0abd14932d4ec4e5fdc6177f2c7cc55c/C of 0abd14932d4ec4e5fdc6177f2c7cc55c into 1717d4eff3cf4343bf0e1fd244864b28(size=11.9 K), total size for store is 11.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 00:55:47,164 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0abd14932d4ec4e5fdc6177f2c7cc55c: 2023-04-16 00:55:47,164 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c., storeName=0abd14932d4ec4e5fdc6177f2c7cc55c/C, priority=12, startTime=1681606547084; duration=0sec 2023-04-16 00:55:47,164 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 00:55:47,165 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:47,165 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:47,165 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:47,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:47,165 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:47,166 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:55:47,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:47,167 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:47,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:55:47,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:47,168 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:55:47,168 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:55:47,168 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:55:47,168 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:55:47,168 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:47,168 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:55:47,169 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:55:47,169 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:55:47,169 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. 2023-04-16 00:55:47,169 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. started... 2023-04-16 00:55:47,169 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 0abd14932d4ec4e5fdc6177f2c7cc55c 3/3 column families, dataSize=127.47 KB heapSize=360.38 KB 2023-04-16 00:55:47,170 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0abd14932d4ec4e5fdc6177f2c7cc55c, store=A 2023-04-16 00:55:47,170 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 00:55:47,170 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0abd14932d4ec4e5fdc6177f2c7cc55c, store=B 2023-04-16 00:55:47,170 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 00:55:47,170 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0abd14932d4ec4e5fdc6177f2c7cc55c, store=C 2023-04-16 00:55:47,170 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 00:55:47,181 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=146 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp/A/be0f2522251c4690a9a527d58be2d4ec 2023-04-16 00:55:47,281 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(9158): Flush requested on 0abd14932d4ec4e5fdc6177f2c7cc55c 2023-04-16 00:55:47,282 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. as already flushing 2023-04-16 00:55:47,320 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:47,320 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57228 deadline: 1681606607312, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:47,320 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:47,320 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57268 deadline: 1681606607313, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:47,321 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:47,321 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57202 deadline: 1681606607313, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:47,327 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:47,327 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57254 deadline: 1681606607322, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:47,341 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:47,341 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57212 deadline: 1681606607337, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:47,421 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:47,421 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57228 deadline: 1681606607421, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:47,598 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=146 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp/B/8c2f5ab27b904c93864bb44be619d59e 2023-04-16 00:55:47,609 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=146 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp/C/cf89f1fc046d48889e7e586e76b4a23b 2023-04-16 00:55:47,613 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp/A/be0f2522251c4690a9a527d58be2d4ec as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/A/be0f2522251c4690a9a527d58be2d4ec 2023-04-16 00:55:47,617 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/A/be0f2522251c4690a9a527d58be2d4ec, entries=150, sequenceid=146, filesize=11.8 K 2023-04-16 00:55:47,618 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp/B/8c2f5ab27b904c93864bb44be619d59e as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/B/8c2f5ab27b904c93864bb44be619d59e 2023-04-16 00:55:47,622 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/B/8c2f5ab27b904c93864bb44be619d59e, entries=150, sequenceid=146, filesize=11.8 K 2023-04-16 00:55:47,622 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp/C/cf89f1fc046d48889e7e586e76b4a23b as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/C/cf89f1fc046d48889e7e586e76b4a23b 2023-04-16 00:55:47,626 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:47,626 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57228 deadline: 1681606607623, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:47,639 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/C/cf89f1fc046d48889e7e586e76b4a23b, entries=150, sequenceid=146, filesize=11.8 K 2023-04-16 00:55:47,639 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~360.33 KB/368976, currentSize=53.67 KB/54960 for 0abd14932d4ec4e5fdc6177f2c7cc55c in 470ms, sequenceid=146, compaction requested=false 2023-04-16 00:55:47,640 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 0abd14932d4ec4e5fdc6177f2c7cc55c: 2023-04-16 00:55:47,640 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. 2023-04-16 00:55:47,640 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:55:47,640 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:55:47,640 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:47,640 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:55:47,640 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:55:47,641 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:47,641 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:47,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:47,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:47,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:47,642 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:47,642 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:55:47,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:47,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:47,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:47,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:47,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:47,643 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:55:47,643 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6bd3efed[Count = 0] remaining members to acquire global barrier 2023-04-16 00:55:47,643 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:55:47,643 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:47,645 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:47,645 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:47,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:47,645 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:55:47,645 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:55:47,645 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:55:47,646 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:47,646 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:55:47,647 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:55:47,647 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:47,647 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:55:47,647 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:47,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:47,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:47,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:47,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:47,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:47,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:47,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:47,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:47,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:47,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:47,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:55:47,650 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:55:47,650 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:55:47,650 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:55:47,650 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:55:47,650 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:55:47,650 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:55:47,654 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:47,654 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:47,654 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:47,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:47,654 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:47,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:47,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:47,655 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:47,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:47,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:47,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:47,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:47,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:47,656 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:47,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:47,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:47,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:47,658 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:47,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:47,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:47,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:47,660 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:47,660 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:47,660 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:47,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:47,661 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:47,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:47,663 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:47,663 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:47,663 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:55:47,663 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:47,663 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:47,663 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:47,663 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:47,663 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:47,663 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:47,663 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:55:47,663 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:55:47,663 DEBUG [Thread-3183] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:55:47,663 DEBUG [Thread-3183] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:55:47,763 DEBUG [Thread-3183] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:55:47,764 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:55:47,765 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:55:47,767 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:55:47,767 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:55:47,767 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:55:47,767 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:47,767 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:55:47,767 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:55:47,767 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:47,767 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:47,768 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:47,768 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:47,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:47,768 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:47,769 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:47,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:47,769 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:55:47,769 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:47,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:55:47,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:47,770 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:55:47,771 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:55:47,771 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:55:47,772 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:55:47,772 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:47,772 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:55:47,772 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:55:47,772 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:55:47,772 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. 2023-04-16 00:55:47,772 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. started... 2023-04-16 00:55:47,772 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 0abd14932d4ec4e5fdc6177f2c7cc55c 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-16 00:55:47,772 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0abd14932d4ec4e5fdc6177f2c7cc55c, store=A 2023-04-16 00:55:47,772 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:55:47,772 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0abd14932d4ec4e5fdc6177f2c7cc55c, store=B 2023-04-16 00:55:47,773 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:55:47,773 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0abd14932d4ec4e5fdc6177f2c7cc55c, store=C 2023-04-16 00:55:47,773 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:55:47,790 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=157 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp/A/6f4c902e8e3b4557999b24c5aa5061e4 2023-04-16 00:55:47,825 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(9158): Flush requested on 0abd14932d4ec4e5fdc6177f2c7cc55c 2023-04-16 00:55:47,825 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. as already flushing 2023-04-16 00:55:47,864 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681606547849, max timestamp=1681606547862Num uniques 859; 2023-04-16 00:55:47,864 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-16 00:55:47,865 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681606547849, max timestamp=1681606547862Num uniques 859; 2023-04-16 00:55:47,865 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-16 00:55:47,865 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681606547849, max timestamp=1681606547862Num uniques 859; 2023-04-16 00:55:47,865 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-16 00:55:47,900 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:47,900 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57254 deadline: 1681606607888, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:47,900 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:47,900 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57268 deadline: 1681606607889, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:47,901 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:47,901 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57202 deadline: 1681606607893, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:47,901 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:47,901 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57212 deadline: 1681606607894, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:47,948 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:47,949 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57228 deadline: 1681606607931, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:48,006 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:48,006 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57254 deadline: 1681606608001, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:48,006 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:48,006 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57268 deadline: 1681606608001, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:48,006 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:48,006 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57202 deadline: 1681606608002, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:48,006 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:48,007 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57212 deadline: 1681606608002, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:48,204 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=157 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp/B/c40a435920d34083a141523f490ea0a6 2023-04-16 00:55:48,215 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:48,215 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57268 deadline: 1681606608207, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:48,215 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:48,216 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57254 deadline: 1681606608207, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:48,216 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:48,216 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57212 deadline: 1681606608207, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:48,216 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:48,216 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57202 deadline: 1681606608208, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:48,457 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:48,457 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57228 deadline: 1681606608453, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:48,520 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:48,521 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57212 deadline: 1681606608517, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:48,521 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:48,521 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57202 deadline: 1681606608517, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:48,521 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:48,521 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57268 deadline: 1681606608517, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:48,522 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:48,522 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57254 deadline: 1681606608518, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:48,617 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=157 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp/C/5976489fbca74f4ca633435e391f12e9 2023-04-16 00:55:48,621 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp/A/6f4c902e8e3b4557999b24c5aa5061e4 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/A/6f4c902e8e3b4557999b24c5aa5061e4 2023-04-16 00:55:48,625 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/A/6f4c902e8e3b4557999b24c5aa5061e4, entries=150, sequenceid=157, filesize=11.8 K 2023-04-16 00:55:48,625 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp/B/c40a435920d34083a141523f490ea0a6 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/B/c40a435920d34083a141523f490ea0a6 2023-04-16 00:55:48,629 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/B/c40a435920d34083a141523f490ea0a6, entries=150, sequenceid=157, filesize=11.8 K 2023-04-16 00:55:48,630 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp/C/5976489fbca74f4ca633435e391f12e9 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/C/5976489fbca74f4ca633435e391f12e9 2023-04-16 00:55:48,633 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/C/5976489fbca74f4ca633435e391f12e9, entries=150, sequenceid=157, filesize=11.8 K 2023-04-16 00:55:48,634 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 0abd14932d4ec4e5fdc6177f2c7cc55c in 862ms, sequenceid=157, compaction requested=true 2023-04-16 00:55:48,634 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 0abd14932d4ec4e5fdc6177f2c7cc55c: 2023-04-16 00:55:48,634 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. 2023-04-16 00:55:48,634 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:55:48,634 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:55:48,634 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:48,634 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:55:48,634 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:55:48,635 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:48,635 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:48,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:48,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:48,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:48,636 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:48,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:48,636 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:55:48,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:48,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:48,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:48,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:48,639 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:55:48,639 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@70b5a5d7[Count = 0] remaining members to acquire global barrier 2023-04-16 00:55:48,639 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:55:48,639 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:48,641 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:48,641 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:48,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:48,641 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:55:48,641 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:55:48,641 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:55:48,641 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:48,641 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:55:48,644 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:48,644 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:55:48,644 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:48,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:48,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:48,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:48,644 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:55:48,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:48,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:48,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:48,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:48,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:48,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:48,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:48,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:55:48,646 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:55:48,646 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:55:48,646 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:55:48,646 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:55:48,646 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:55:48,646 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:55:48,648 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:48,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:48,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:48,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:48,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:48,648 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:48,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:48,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:48,649 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:48,649 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:48,649 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:48,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:48,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:48,650 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:48,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:48,650 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:48,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:48,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:48,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:48,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:48,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:48,656 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:48,656 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:48,656 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:48,656 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:48,656 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:48,656 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:48,656 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:48,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:48,656 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:48,656 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:48,656 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:48,656 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:48,656 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:55:48,656 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:55:48,656 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:55:48,656 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:48,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:48,657 DEBUG [Thread-3183] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:55:48,657 DEBUG [Thread-3183] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:55:48,757 DEBUG [Thread-3183] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:55:48,757 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:55:48,759 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:55:48,760 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:55:48,760 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:55:48,760 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:55:48,760 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:48,760 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:55:48,760 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:55:48,760 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:48,760 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:48,762 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:48,762 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:48,762 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:48,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:48,762 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:48,762 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:55:48,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:48,762 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:48,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:55:48,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:48,763 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:55:48,763 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:55:48,763 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:55:48,763 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:55:48,763 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:48,763 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:55:48,763 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:55:48,763 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:55:48,764 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. 2023-04-16 00:55:48,764 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. started... 2023-04-16 00:55:48,764 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 0abd14932d4ec4e5fdc6177f2c7cc55c 3/3 column families, dataSize=127.47 KB heapSize=365.06 KB 2023-04-16 00:55:48,764 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0abd14932d4ec4e5fdc6177f2c7cc55c, store=A 2023-04-16 00:55:48,764 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 00:55:48,764 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0abd14932d4ec4e5fdc6177f2c7cc55c, store=B 2023-04-16 00:55:48,764 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 00:55:48,764 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0abd14932d4ec4e5fdc6177f2c7cc55c, store=C 2023-04-16 00:55:48,764 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 00:55:48,771 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=193 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp/A/bfea04ef6d5c40ab826f04ad1a891b09 2023-04-16 00:55:48,782 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=193 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp/B/0aaf263ff33f44608cd1ff27a4f1c8de 2023-04-16 00:55:48,795 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=193 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp/C/7e9416008d8e4d0785b8177e58000f15 2023-04-16 00:55:48,799 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp/A/bfea04ef6d5c40ab826f04ad1a891b09 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/A/bfea04ef6d5c40ab826f04ad1a891b09 2023-04-16 00:55:48,805 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/A/bfea04ef6d5c40ab826f04ad1a891b09, entries=150, sequenceid=193, filesize=11.8 K 2023-04-16 00:55:48,805 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp/B/0aaf263ff33f44608cd1ff27a4f1c8de as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/B/0aaf263ff33f44608cd1ff27a4f1c8de 2023-04-16 00:55:48,809 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/B/0aaf263ff33f44608cd1ff27a4f1c8de, entries=150, sequenceid=193, filesize=11.8 K 2023-04-16 00:55:48,810 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp/C/7e9416008d8e4d0785b8177e58000f15 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/C/7e9416008d8e4d0785b8177e58000f15 2023-04-16 00:55:48,814 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/C/7e9416008d8e4d0785b8177e58000f15, entries=150, sequenceid=193, filesize=11.8 K 2023-04-16 00:55:48,815 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~365.02 KB/373776, currentSize=0 B/0 for 0abd14932d4ec4e5fdc6177f2c7cc55c in 51ms, sequenceid=193, compaction requested=true 2023-04-16 00:55:48,815 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 0abd14932d4ec4e5fdc6177f2c7cc55c: 2023-04-16 00:55:48,815 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. 2023-04-16 00:55:48,815 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:55:48,815 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:55:48,815 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:48,815 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:55:48,815 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:55:48,817 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:48,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:48,817 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:48,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:48,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:48,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:48,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:48,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:48,818 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:48,818 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:55:48,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:48,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:48,819 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:55:48,819 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7f0eb26b[Count = 0] remaining members to acquire global barrier 2023-04-16 00:55:48,819 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:55:48,819 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:48,820 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:48,820 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:48,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:48,820 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:55:48,820 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:48,820 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:55:48,820 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:55:48,820 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:55:48,823 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:55:48,823 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:48,823 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:55:48,823 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:48,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:48,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:48,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:48,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:48,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:48,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:48,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:48,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:48,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:48,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:48,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:55:48,826 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:55:48,826 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:55:48,826 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:55:48,826 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:55:48,826 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:55:48,826 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:55:48,828 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:48,828 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:48,828 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:48,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:48,828 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:48,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:48,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:48,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:48,828 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:48,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:48,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:48,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:48,829 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:48,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:48,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:48,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:48,830 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:48,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:48,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:48,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:48,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:48,833 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:48,833 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:55:48,833 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:55:48,833 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:55:48,833 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:48,833 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:48,833 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:48,833 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:48,834 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:48,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:48,834 DEBUG [Thread-3183] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:55:48,834 DEBUG [Thread-3183] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:55:48,834 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:48,834 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:48,834 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:48,834 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:48,834 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:48,834 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:48,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:48,934 DEBUG [Thread-3183] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:55:48,934 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:55:48,936 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:55:48,937 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:55:48,937 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:55:48,937 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:55:48,937 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:48,937 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:55:48,937 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:55:48,938 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:48,938 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:48,939 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:48,939 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:48,939 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:48,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:48,939 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:48,939 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:55:48,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:48,940 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:48,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:55:48,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:48,940 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:55:48,940 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:55:48,940 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:55:48,940 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:55:48,940 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:48,940 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:55:48,941 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:55:48,941 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:55:48,941 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. 2023-04-16 00:55:48,941 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. started... 2023-04-16 00:55:48,941 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 0abd14932d4ec4e5fdc6177f2c7cc55c: 2023-04-16 00:55:48,941 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. 2023-04-16 00:55:48,942 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:55:48,942 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:55:48,942 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:48,942 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:55:48,942 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:55:48,945 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:48,945 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:48,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:48,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:48,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:48,945 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:48,945 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:55:48,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:48,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:48,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:48,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:48,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:48,947 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:55:48,947 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2ed457ba[Count = 0] remaining members to acquire global barrier 2023-04-16 00:55:48,947 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:55:48,947 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:48,948 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:48,948 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:48,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:48,948 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:48,949 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:55:48,949 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:55:48,949 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:55:48,949 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:55:48,950 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:55:48,950 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:48,950 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:48,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:48,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:48,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:48,950 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:55:48,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:48,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:48,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:48,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:48,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:48,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:48,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:48,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:55:48,953 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:55:48,953 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:55:48,953 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:55:48,953 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:55:48,953 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:55:48,953 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:55:48,955 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:48,955 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:48,955 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:48,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:48,955 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:48,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:48,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:48,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:48,955 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:48,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:48,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:48,956 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:48,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:48,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:48,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:48,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:48,957 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:48,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:48,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:48,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:48,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:48,960 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:48,960 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:48,961 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:48,961 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:48,961 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:48,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:48,961 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:48,961 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:55:48,961 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:55:48,961 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:48,961 DEBUG [Thread-3183] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:55:48,961 DEBUG [Thread-3183] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:55:48,961 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:48,961 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:48,961 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:48,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:48,961 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:48,961 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:55:48,961 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:49,033 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(9158): Flush requested on 0abd14932d4ec4e5fdc6177f2c7cc55c 2023-04-16 00:55:49,033 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 0abd14932d4ec4e5fdc6177f2c7cc55c 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-16 00:55:49,033 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0abd14932d4ec4e5fdc6177f2c7cc55c, store=A 2023-04-16 00:55:49,033 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:55:49,033 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0abd14932d4ec4e5fdc6177f2c7cc55c, store=B 2023-04-16 00:55:49,033 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:55:49,033 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0abd14932d4ec4e5fdc6177f2c7cc55c, store=C 2023-04-16 00:55:49,033 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:55:49,061 DEBUG [Thread-3183] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:55:49,062 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:55:49,064 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:55:49,065 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:55:49,065 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:55:49,065 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:55:49,065 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:49,065 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:55:49,065 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:55:49,066 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:49,066 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:49,067 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:49,067 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:49,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:49,067 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:49,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:49,067 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:49,067 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:55:49,068 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:49,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:55:49,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:49,068 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:55:49,068 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:55:49,068 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:55:49,068 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:55:49,069 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:49,069 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:55:49,069 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:55:49,069 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:55:49,069 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. 2023-04-16 00:55:49,069 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. started... 2023-04-16 00:55:49,069 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. as already flushing 2023-04-16 00:55:49,069 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. 2023-04-16 00:55:49,069 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:55:49,069 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:55:49,069 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:49,069 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:55:49,069 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:55:49,070 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:49,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:49,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:49,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:49,071 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:49,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:49,071 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:49,071 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:55:49,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:49,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:49,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:49,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:49,073 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:55:49,073 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5a8c790a[Count = 0] remaining members to acquire global barrier 2023-04-16 00:55:49,073 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:55:49,073 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:49,074 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:49,074 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:49,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:49,074 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:55:49,074 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:55:49,074 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:49,074 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:55:49,074 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:55:49,077 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:49,077 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:55:49,077 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:49,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:49,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:49,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:49,077 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:55:49,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:49,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:49,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:49,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:49,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:49,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:49,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:49,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:55:49,081 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:55:49,081 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:55:49,082 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:55:49,082 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:55:49,082 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:55:49,082 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:55:49,083 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:49,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:49,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:49,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:49,083 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:49,083 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:49,083 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:49,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:49,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:49,083 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:49,083 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:49,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:49,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:49,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:49,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:49,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:49,085 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:49,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:49,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:49,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:49,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:49,089 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:49,089 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:49,089 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:49,089 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:49,089 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:49,089 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:49,089 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:49,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:49,089 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:49,089 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:55:49,089 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:55:49,089 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:49,089 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:49,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:49,089 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:49,089 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:55:49,090 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:49,089 DEBUG [Thread-3183] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:55:49,090 DEBUG [Thread-3183] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:55:49,095 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11854, totalHeapSize=27176, min timestamp=1681606549071, max timestamp=1681606549092Num uniques 859; 2023-04-16 00:55:49,095 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=11854, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-16 00:55:49,095 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11854, totalHeapSize=27176, min timestamp=1681606549071, max timestamp=1681606549092Num uniques 859; 2023-04-16 00:55:49,095 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=11854, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-16 00:55:49,096 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11854, totalHeapSize=27176, min timestamp=1681606549071, max timestamp=1681606549092Num uniques 859; 2023-04-16 00:55:49,096 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=11854, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-16 00:55:49,134 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:49,134 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57212 deadline: 1681606609125, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:49,135 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:49,135 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57254 deadline: 1681606609130, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:49,145 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:49,146 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57202 deadline: 1681606609133, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:49,146 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:49,146 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57268 deadline: 1681606609134, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:49,190 DEBUG [Thread-3183] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:55:49,190 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:55:49,192 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:55:49,193 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:55:49,193 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:55:49,193 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:55:49,193 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:49,193 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:55:49,193 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:55:49,193 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:49,193 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:49,195 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:49,195 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:49,195 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:49,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:49,196 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:49,196 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:55:49,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:49,196 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:49,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:55:49,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:49,196 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:55:49,197 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:55:49,197 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:55:49,197 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:55:49,197 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:49,197 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:55:49,197 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:55:49,197 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:55:49,197 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. 2023-04-16 00:55:49,197 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. started... 2023-04-16 00:55:49,197 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. as already flushing 2023-04-16 00:55:49,197 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. 2023-04-16 00:55:49,197 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:55:49,197 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:55:49,197 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:49,197 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:55:49,197 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:55:49,198 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:49,198 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:49,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:49,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:49,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:49,199 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:49,199 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:55:49,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:49,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:49,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:49,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:49,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:49,200 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:55:49,200 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@74ae491[Count = 0] remaining members to acquire global barrier 2023-04-16 00:55:49,200 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:55:49,200 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:49,201 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:49,201 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:49,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:49,201 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:55:49,201 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:55:49,201 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:55:49,202 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:49,202 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:55:49,203 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:49,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:49,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:49,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:49,203 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:55:49,203 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:49,203 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:55:49,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:49,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:49,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:49,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:49,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:49,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:49,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:49,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:55:49,205 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:55:49,205 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:55:49,205 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:55:49,205 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:55:49,205 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:55:49,205 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:55:49,207 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:49,207 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:49,207 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:49,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:49,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:49,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:49,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:49,207 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:49,207 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:49,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:49,208 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:49,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:49,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:49,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:49,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:49,210 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:49,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:49,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:49,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:49,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:49,211 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:49,211 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:49,211 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:49,211 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:49,211 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:49,212 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:49,212 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:55:49,212 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:55:49,212 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:49,212 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:49,212 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:49,212 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:49,212 DEBUG [Thread-3183] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:55:49,212 DEBUG [Thread-3183] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:55:49,212 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:49,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:49,212 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:49,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:49,212 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:55:49,241 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:49,241 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57212 deadline: 1681606609235, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:49,241 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:49,241 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57254 deadline: 1681606609236, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:49,253 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:49,253 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57202 deadline: 1681606609247, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:49,253 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:49,253 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57268 deadline: 1681606609247, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:49,312 DEBUG [Thread-3183] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:55:49,313 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:55:49,315 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:55:49,315 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:55:49,316 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:55:49,316 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:55:49,316 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:49,316 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:55:49,316 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:55:49,316 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:49,316 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:49,317 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:49,317 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:49,317 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:49,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:49,318 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:49,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:49,318 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:55:49,318 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:49,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:55:49,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:49,318 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:55:49,319 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:55:49,319 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:55:49,319 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:55:49,319 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:49,319 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:55:49,319 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:55:49,319 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:55:49,319 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. 2023-04-16 00:55:49,319 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. started... 2023-04-16 00:55:49,319 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. as already flushing 2023-04-16 00:55:49,319 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. 2023-04-16 00:55:49,319 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:55:49,319 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:55:49,319 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:49,319 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:55:49,319 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:55:49,321 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:49,321 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:49,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:49,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:49,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:49,321 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:49,321 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:55:49,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:49,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:49,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:49,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:49,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:49,322 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:55:49,322 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@ddd83f9[Count = 0] remaining members to acquire global barrier 2023-04-16 00:55:49,322 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:55:49,322 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:49,324 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:49,324 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:49,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:49,324 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:55:49,324 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:49,324 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:55:49,324 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:55:49,324 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:55:49,326 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:55:49,326 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:49,326 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:49,326 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:55:49,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:49,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:49,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:49,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:49,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:49,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:49,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:49,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:49,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:49,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:49,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:55:49,328 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:55:49,328 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:55:49,328 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:55:49,328 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:55:49,328 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:55:49,328 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:55:49,329 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:49,329 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:49,329 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:49,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:49,329 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:49,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:49,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:49,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:49,330 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:49,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:49,330 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:49,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:49,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:49,331 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:49,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:49,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:49,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:49,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:49,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:49,334 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:49,334 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:49,334 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:49,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:49,334 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:49,334 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:49,334 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:55:49,334 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:55:49,334 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:49,334 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:55:49,334 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:49,334 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:49,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:49,334 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:49,334 DEBUG [Thread-3183] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:55:49,334 DEBUG [Thread-3183] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:55:49,334 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:49,335 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:49,335 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:49,435 DEBUG [Thread-3183] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:55:49,435 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:55:49,437 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:55:49,438 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:55:49,438 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:55:49,438 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:55:49,438 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:49,438 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:55:49,438 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:55:49,438 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:49,439 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:49,442 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:49,442 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:49,442 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:49,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:49,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:49,442 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:49,443 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:55:49,443 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:49,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:55:49,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:49,443 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=204 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp/A/6d59006731764706bee1a03812236949 2023-04-16 00:55:49,443 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:55:49,444 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:55:49,444 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:55:49,444 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:55:49,444 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:49,444 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:55:49,444 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:55:49,444 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:55:49,444 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. 2023-04-16 00:55:49,444 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. started... 2023-04-16 00:55:49,444 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. as already flushing 2023-04-16 00:55:49,444 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. 2023-04-16 00:55:49,444 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:49,444 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57254 deadline: 1681606609442, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:49,444 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:55:49,445 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:55:49,445 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:49,445 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:55:49,445 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:55:49,446 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:49,446 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:49,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:49,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:49,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:49,446 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:49,446 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:55:49,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:49,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:49,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:49,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:49,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:49,450 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:55:49,450 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1387c7f[Count = 0] remaining members to acquire global barrier 2023-04-16 00:55:49,450 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:55:49,450 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:49,450 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:49,451 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57212 deadline: 1681606609443, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:49,451 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:49,451 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:49,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:49,451 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:55:49,451 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:55:49,451 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:55:49,453 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:49,453 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:55:49,453 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:49,453 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:55:49,453 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:55:49,453 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:55:49,453 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:55:49,453 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:55:49,453 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:55:49,453 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:55:49,453 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:55:49,454 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=204 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp/B/4cb6a22b1d824413a2892ccf35d20cd0 2023-04-16 00:55:49,458 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:49,458 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:49,458 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:49,458 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:49,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:49,459 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:49,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:49,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:49,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:49,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:49,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:49,459 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:49,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:49,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:49,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:49,461 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:49,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:49,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:49,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:49,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:49,463 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:49,463 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:49,463 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:49,463 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:49,463 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:49,463 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:49,463 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:49,463 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:49,463 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:55:49,463 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:55:49,463 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:49,463 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:55:49,463 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:49,463 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:49,463 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:49,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:49,464 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57202 deadline: 1681606609456, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:49,464 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:49,464 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:49,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:49,464 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57268 deadline: 1681606609456, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:49,464 DEBUG [Thread-3183] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:55:49,465 DEBUG [Thread-3183] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:55:49,465 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:49,465 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57228 deadline: 1681606609461, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:49,466 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=204 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp/C/7f3c94448c75443b9243c01b1efa9241 2023-04-16 00:55:49,469 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp/A/6d59006731764706bee1a03812236949 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/A/6d59006731764706bee1a03812236949 2023-04-16 00:55:49,472 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/A/6d59006731764706bee1a03812236949, entries=200, sequenceid=204, filesize=14.1 K 2023-04-16 00:55:49,472 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp/B/4cb6a22b1d824413a2892ccf35d20cd0 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/B/4cb6a22b1d824413a2892ccf35d20cd0 2023-04-16 00:55:49,475 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/B/4cb6a22b1d824413a2892ccf35d20cd0, entries=150, sequenceid=204, filesize=11.8 K 2023-04-16 00:55:49,476 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp/C/7f3c94448c75443b9243c01b1efa9241 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/C/7f3c94448c75443b9243c01b1efa9241 2023-04-16 00:55:49,480 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/C/7f3c94448c75443b9243c01b1efa9241, entries=150, sequenceid=204, filesize=11.8 K 2023-04-16 00:55:49,481 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 0abd14932d4ec4e5fdc6177f2c7cc55c in 448ms, sequenceid=204, compaction requested=true 2023-04-16 00:55:49,481 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 0abd14932d4ec4e5fdc6177f2c7cc55c: 2023-04-16 00:55:49,481 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 00:55:49,481 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-16 00:55:49,481 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-16 00:55:49,481 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 00:55:49,481 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 00:55:49,482 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 62694 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-16 00:55:49,483 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HStore(1912): 0abd14932d4ec4e5fdc6177f2c7cc55c/A is initiating minor compaction (all files) 2023-04-16 00:55:49,483 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 0abd14932d4ec4e5fdc6177f2c7cc55c/A in TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. 2023-04-16 00:55:49,483 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/A/7296aeea67a740a3baa8203cdefc1ba8, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/A/be0f2522251c4690a9a527d58be2d4ec, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/A/6f4c902e8e3b4557999b24c5aa5061e4, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/A/bfea04ef6d5c40ab826f04ad1a891b09, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/A/6d59006731764706bee1a03812236949] into tmpdir=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp, totalSize=61.2 K 2023-04-16 00:55:49,483 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 60304 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-16 00:55:49,483 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.Compactor(207): Compacting 7296aeea67a740a3baa8203cdefc1ba8, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=108, earliestPutTs=1681606546611 2023-04-16 00:55:49,483 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HStore(1912): 0abd14932d4ec4e5fdc6177f2c7cc55c/B is initiating minor compaction (all files) 2023-04-16 00:55:49,483 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 0abd14932d4ec4e5fdc6177f2c7cc55c/B in TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. 2023-04-16 00:55:49,483 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/B/51638d2847714f59a9001c13cc9b1ddf, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/B/8c2f5ab27b904c93864bb44be619d59e, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/B/c40a435920d34083a141523f490ea0a6, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/B/0aaf263ff33f44608cd1ff27a4f1c8de, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/B/4cb6a22b1d824413a2892ccf35d20cd0] into tmpdir=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp, totalSize=58.9 K 2023-04-16 00:55:49,484 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.Compactor(207): Compacting be0f2522251c4690a9a527d58be2d4ec, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=146, earliestPutTs=1681606546681 2023-04-16 00:55:49,484 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting 51638d2847714f59a9001c13cc9b1ddf, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=108, earliestPutTs=1681606546611 2023-04-16 00:55:49,484 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.Compactor(207): Compacting 6f4c902e8e3b4557999b24c5aa5061e4, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=157, earliestPutTs=1681606547294 2023-04-16 00:55:49,484 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting 8c2f5ab27b904c93864bb44be619d59e, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=146, earliestPutTs=1681606546681 2023-04-16 00:55:49,484 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.Compactor(207): Compacting bfea04ef6d5c40ab826f04ad1a891b09, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=193, earliestPutTs=1681606547886 2023-04-16 00:55:49,484 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting c40a435920d34083a141523f490ea0a6, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=157, earliestPutTs=1681606547294 2023-04-16 00:55:49,485 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.Compactor(207): Compacting 6d59006731764706bee1a03812236949, keycount=200, bloomtype=ROW, size=14.1 K, encoding=NONE, compression=NONE, seqNum=204, earliestPutTs=1681606549027 2023-04-16 00:55:49,485 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting 0aaf263ff33f44608cd1ff27a4f1c8de, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=193, earliestPutTs=1681606547886 2023-04-16 00:55:49,485 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting 4cb6a22b1d824413a2892ccf35d20cd0, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=204, earliestPutTs=1681606549028 2023-04-16 00:55:49,493 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] throttle.PressureAwareThroughputController(145): 0abd14932d4ec4e5fdc6177f2c7cc55c#A#compaction#444 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-16 00:55:49,496 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] throttle.PressureAwareThroughputController(145): 0abd14932d4ec4e5fdc6177f2c7cc55c#B#compaction#445 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-16 00:55:49,565 DEBUG [Thread-3183] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:55:49,565 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:55:49,567 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:55:49,568 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:55:49,568 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:55:49,568 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:55:49,568 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:49,568 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:55:49,568 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:55:49,569 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:49,569 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:49,570 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:49,570 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:49,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:49,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:49,571 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:49,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:55:49,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:49,571 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:55:49,571 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:49,572 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:49,572 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:55:49,572 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:55:49,572 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:55:49,572 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:55:49,572 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:49,572 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:55:49,573 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:55:49,573 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:55:49,573 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. 2023-04-16 00:55:49,573 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. started... 2023-04-16 00:55:49,573 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 0abd14932d4ec4e5fdc6177f2c7cc55c 3/3 column families, dataSize=134.18 KB heapSize=375.61 KB 2023-04-16 00:55:49,573 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0abd14932d4ec4e5fdc6177f2c7cc55c, store=A 2023-04-16 00:55:49,573 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 00:55:49,573 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0abd14932d4ec4e5fdc6177f2c7cc55c, store=B 2023-04-16 00:55:49,574 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 00:55:49,574 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0abd14932d4ec4e5fdc6177f2c7cc55c, store=C 2023-04-16 00:55:49,574 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 00:55:49,581 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=239 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp/A/e8351b101df84b41b3666061b9a9d395 2023-04-16 00:55:49,596 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=239 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp/B/4205e1d340c241bfb527f56828df10e3 2023-04-16 00:55:49,607 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=239 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp/C/f4795edb7d7d4d3aac22ad9a99bdc211 2023-04-16 00:55:49,611 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp/A/e8351b101df84b41b3666061b9a9d395 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/A/e8351b101df84b41b3666061b9a9d395 2023-04-16 00:55:49,615 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/A/e8351b101df84b41b3666061b9a9d395, entries=150, sequenceid=239, filesize=11.8 K 2023-04-16 00:55:49,616 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp/B/4205e1d340c241bfb527f56828df10e3 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/B/4205e1d340c241bfb527f56828df10e3 2023-04-16 00:55:49,619 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/B/4205e1d340c241bfb527f56828df10e3, entries=150, sequenceid=239, filesize=11.8 K 2023-04-16 00:55:49,619 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp/C/f4795edb7d7d4d3aac22ad9a99bdc211 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/C/f4795edb7d7d4d3aac22ad9a99bdc211 2023-04-16 00:55:49,627 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/C/f4795edb7d7d4d3aac22ad9a99bdc211, entries=150, sequenceid=239, filesize=11.8 K 2023-04-16 00:55:49,628 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~375.56 KB/384576, currentSize=0 B/0 for 0abd14932d4ec4e5fdc6177f2c7cc55c in 55ms, sequenceid=239, compaction requested=true 2023-04-16 00:55:49,628 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 0abd14932d4ec4e5fdc6177f2c7cc55c: 2023-04-16 00:55:49,628 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. 2023-04-16 00:55:49,628 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:55:49,628 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:55:49,628 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:49,628 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:55:49,628 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:55:49,633 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:49,633 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:49,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:49,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:49,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:49,634 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:49,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:49,634 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:55:49,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:49,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:49,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:49,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:49,635 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:55:49,635 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@38c5117b[Count = 0] remaining members to acquire global barrier 2023-04-16 00:55:49,635 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:55:49,635 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:49,638 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:49,638 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:49,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:49,638 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:55:49,638 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:55:49,638 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:55:49,638 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:49,638 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:55:49,639 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:55:49,639 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:49,639 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:49,639 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:55:49,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:49,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:49,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:49,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:49,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:49,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:49,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:49,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:49,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:49,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:49,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:55:49,641 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:55:49,641 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:55:49,641 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:55:49,642 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:55:49,642 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:55:49,642 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:55:49,645 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:49,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:49,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:49,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:49,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:49,646 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:49,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:49,646 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:49,646 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:49,646 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:49,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:49,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:49,646 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:49,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:49,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:49,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:49,647 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:49,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:49,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:49,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:49,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:49,649 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:49,649 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:49,649 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:49,649 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:55:49,649 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:55:49,649 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:49,649 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:55:49,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:49,649 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:49,649 DEBUG [Thread-3183] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:55:49,649 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:49,649 DEBUG [Thread-3183] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:55:49,649 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:49,649 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:49,649 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:49,649 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:49,650 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:49,650 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:49,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:49,749 DEBUG [Thread-3183] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:55:49,750 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:55:49,751 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:55:49,753 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:55:49,753 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:55:49,753 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:55:49,753 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:49,753 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:55:49,753 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:55:49,753 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:49,753 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:49,754 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:49,755 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:49,755 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:49,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:49,755 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:49,755 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:55:49,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:49,755 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:49,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:55:49,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:49,756 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:55:49,756 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:55:49,756 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:55:49,756 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:55:49,756 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:49,756 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:55:49,756 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:55:49,756 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:55:49,756 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. 2023-04-16 00:55:49,757 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. started... 2023-04-16 00:55:49,757 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 0abd14932d4ec4e5fdc6177f2c7cc55c 3/3 column families, dataSize=20.13 KB heapSize=60.52 KB 2023-04-16 00:55:49,757 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0abd14932d4ec4e5fdc6177f2c7cc55c, store=A 2023-04-16 00:55:49,757 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:55:49,757 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0abd14932d4ec4e5fdc6177f2c7cc55c, store=B 2023-04-16 00:55:49,757 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:55:49,757 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0abd14932d4ec4e5fdc6177f2c7cc55c, store=C 2023-04-16 00:55:49,757 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:55:49,764 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=245 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp/A/b627f7424e304af0a35e600740b3395d 2023-04-16 00:55:49,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(9158): Flush requested on 0abd14932d4ec4e5fdc6177f2c7cc55c 2023-04-16 00:55:49,773 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. as already flushing 2023-04-16 00:55:49,774 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=245 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp/B/ea3738a001ff4d7494adfc7f72a3b985 2023-04-16 00:55:49,804 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681606549785, max timestamp=1681606549803Num uniques 859; 2023-04-16 00:55:49,804 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-16 00:55:49,805 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681606549785, max timestamp=1681606549803Num uniques 859; 2023-04-16 00:55:49,805 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-16 00:55:49,806 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681606549785, max timestamp=1681606549803Num uniques 859; 2023-04-16 00:55:49,806 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-16 00:55:49,853 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=318, cellSize=14558, totalHeapSize=33312, min timestamp=1681606549819, max timestamp=1681606549850Num uniques 1018; 2023-04-16 00:55:49,853 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=318, cellSize=14558, totalHeapSize=33312, min timestamp=1681606549819, max timestamp=1681606549850Num uniques 1018; 2023-04-16 00:55:49,853 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=318, cellSize=14558, totalHeapSize=33312, min timestamp=1681606549819, max timestamp=1681606549850Num uniques 1018; 2023-04-16 00:55:49,853 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=14558, suffix heap size=133840,new segment heap size=33312 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=318 2023-04-16 00:55:49,853 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=14558, suffix heap size=133840,new segment heap size=33312 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=318 2023-04-16 00:55:49,853 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=14558, suffix heap size=133840,new segment heap size=33312 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=318 2023-04-16 00:55:49,914 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp/A/cba347c854c54e6091c926834313b1c9 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/A/cba347c854c54e6091c926834313b1c9 2023-04-16 00:55:49,914 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp/B/544c2a6ab4d04325accff040f2e4827d as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/B/544c2a6ab4d04325accff040f2e4827d 2023-04-16 00:55:49,930 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 0abd14932d4ec4e5fdc6177f2c7cc55c/B of 0abd14932d4ec4e5fdc6177f2c7cc55c into 544c2a6ab4d04325accff040f2e4827d(size=12.2 K), total size for store is 23.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 00:55:49,930 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0abd14932d4ec4e5fdc6177f2c7cc55c: 2023-04-16 00:55:49,930 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c., storeName=0abd14932d4ec4e5fdc6177f2c7cc55c/B, priority=11, startTime=1681606549481; duration=0sec 2023-04-16 00:55:49,931 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 00:55:49,931 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 0abd14932d4ec4e5fdc6177f2c7cc55c/A of 0abd14932d4ec4e5fdc6177f2c7cc55c into cba347c854c54e6091c926834313b1c9(size=12.2 K), total size for store is 23.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 00:55:49,931 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0abd14932d4ec4e5fdc6177f2c7cc55c: 2023-04-16 00:55:49,931 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c., storeName=0abd14932d4ec4e5fdc6177f2c7cc55c/A, priority=11, startTime=1681606549481; duration=0sec 2023-04-16 00:55:49,931 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:49,931 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-16 00:55:49,931 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 00:55:49,932 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57202 deadline: 1681606609896, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:49,932 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10382, totalHeapSize=23848, min timestamp=1681606549881, max timestamp=1681606549916Num uniques 1177; 2023-04-16 00:55:49,932 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=53892, new segment data size=10382, suffix heap size=150376,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1177, new segment cells count=227 2023-04-16 00:55:49,933 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10382, totalHeapSize=23848, min timestamp=1681606549881, max timestamp=1681606549916Num uniques 1177; 2023-04-16 00:55:49,933 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=53892, new segment data size=10382, suffix heap size=150376,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1177, new segment cells count=227 2023-04-16 00:55:49,933 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1681606549881, max timestamp=1681606549916Num uniques 1177; 2023-04-16 00:55:49,933 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=53902, new segment data size=10392, suffix heap size=150376,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1177, new segment cells count=227 2023-04-16 00:55:49,934 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 72340 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-16 00:55:49,934 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HStore(1912): 0abd14932d4ec4e5fdc6177f2c7cc55c/C is initiating minor compaction (all files) 2023-04-16 00:55:49,934 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 0abd14932d4ec4e5fdc6177f2c7cc55c/C in TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. 2023-04-16 00:55:49,934 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/C/1717d4eff3cf4343bf0e1fd244864b28, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/C/cf89f1fc046d48889e7e586e76b4a23b, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/C/5976489fbca74f4ca633435e391f12e9, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/C/7e9416008d8e4d0785b8177e58000f15, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/C/7f3c94448c75443b9243c01b1efa9241, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/C/f4795edb7d7d4d3aac22ad9a99bdc211] into tmpdir=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp, totalSize=70.6 K 2023-04-16 00:55:49,936 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting 1717d4eff3cf4343bf0e1fd244864b28, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=108, earliestPutTs=1681606546611 2023-04-16 00:55:49,936 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting cf89f1fc046d48889e7e586e76b4a23b, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=146, earliestPutTs=1681606546681 2023-04-16 00:55:49,937 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting 5976489fbca74f4ca633435e391f12e9, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=157, earliestPutTs=1681606547294 2023-04-16 00:55:49,937 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting 7e9416008d8e4d0785b8177e58000f15, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=193, earliestPutTs=1681606547886 2023-04-16 00:55:49,937 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting 7f3c94448c75443b9243c01b1efa9241, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=204, earliestPutTs=1681606549028 2023-04-16 00:55:49,937 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting f4795edb7d7d4d3aac22ad9a99bdc211, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=239, earliestPutTs=1681606549124 2023-04-16 00:55:49,946 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] throttle.PressureAwareThroughputController(145): 0abd14932d4ec4e5fdc6177f2c7cc55c#C#compaction#452 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-16 00:55:49,957 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp/C/14da418ff71c41c69a3febacebd88a8b as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/C/14da418ff71c41c69a3febacebd88a8b 2023-04-16 00:55:49,960 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 0abd14932d4ec4e5fdc6177f2c7cc55c/C of 0abd14932d4ec4e5fdc6177f2c7cc55c into 14da418ff71c41c69a3febacebd88a8b(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 00:55:49,960 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0abd14932d4ec4e5fdc6177f2c7cc55c: 2023-04-16 00:55:49,960 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c., storeName=0abd14932d4ec4e5fdc6177f2c7cc55c/C, priority=10, startTime=1681606549481; duration=0sec 2023-04-16 00:55:49,960 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 00:55:49,985 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1681606549963, max timestamp=1681606549981Num uniques 1086; 2023-04-16 00:55:49,985 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=8516, suffix heap size=140912,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=186 2023-04-16 00:55:49,986 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1681606549963, max timestamp=1681606549981Num uniques 1086; 2023-04-16 00:55:49,986 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=10806, suffix heap size=140912,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=236 2023-04-16 00:55:49,986 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1681606549963, max timestamp=1681606549981Num uniques 1086; 2023-04-16 00:55:49,986 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=8516, suffix heap size=140912,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=186 2023-04-16 00:55:50,021 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=295, cellSize=13510, totalHeapSize=30920, min timestamp=1681606550007, max timestamp=1681606550018Num uniques 1045; 2023-04-16 00:55:50,021 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=13510, suffix heap size=136648,new segment heap size=30920 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=295 2023-04-16 00:55:50,021 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=295, cellSize=13510, totalHeapSize=30920, min timestamp=1681606550007, max timestamp=1681606550018Num uniques 1095; 2023-04-16 00:55:50,022 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=50150, new segment data size=13510, suffix heap size=141848,new segment heap size=30920 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1095, new segment cells count=295 2023-04-16 00:55:50,022 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=295, cellSize=13510, totalHeapSize=30920, min timestamp=1681606550007, max timestamp=1681606550018Num uniques 1045; 2023-04-16 00:55:50,022 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=13510, suffix heap size=136648,new segment heap size=30920 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=295 2023-04-16 00:55:50,057 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:50,057 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57254 deadline: 1681606610052, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:50,058 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=304, cellSize=13920, totalHeapSize=31856, min timestamp=1681606550038, max timestamp=1681606550056Num uniques 1154; 2023-04-16 00:55:50,058 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=52850, new segment data size=13920, suffix heap size=147984,new segment heap size=31856 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1154, new segment cells count=304 2023-04-16 00:55:50,058 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=304, cellSize=13920, totalHeapSize=31856, min timestamp=1681606550038, max timestamp=1681606550056Num uniques 1154; 2023-04-16 00:55:50,059 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=304, cellSize=13920, totalHeapSize=31856, min timestamp=1681606550038, max timestamp=1681606550056Num uniques 1154; 2023-04-16 00:55:50,059 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=52850, new segment data size=13920, suffix heap size=147984,new segment heap size=31856 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1154, new segment cells count=304 2023-04-16 00:55:50,059 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=52850, new segment data size=13920, suffix heap size=147984,new segment heap size=31856 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1154, new segment cells count=304 2023-04-16 00:55:50,099 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=263, cellSize=12038, totalHeapSize=27592, min timestamp=1681606550086, max timestamp=1681606550097Num uniques 1163; 2023-04-16 00:55:50,099 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=53258, new segment data size=12038, suffix heap size=148920,new segment heap size=27592 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1163, new segment cells count=263 2023-04-16 00:55:50,100 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=263, cellSize=12038, totalHeapSize=27592, min timestamp=1681606550086, max timestamp=1681606550097Num uniques 1163; 2023-04-16 00:55:50,100 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=53258, new segment data size=12038, suffix heap size=148920,new segment heap size=27592 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1163, new segment cells count=263 2023-04-16 00:55:50,100 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=263, cellSize=12038, totalHeapSize=27592, min timestamp=1681606550086, max timestamp=1681606550097Num uniques 1163; 2023-04-16 00:55:50,100 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=53258, new segment data size=12038, suffix heap size=148920,new segment heap size=27592 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1163, new segment cells count=263 2023-04-16 00:55:50,143 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=222, cellSize=10162, totalHeapSize=23328, min timestamp=1681606550125, max timestamp=1681606550140Num uniques 1122; 2023-04-16 00:55:50,143 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=51382, new segment data size=10162, suffix heap size=144656,new segment heap size=23328 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1122, new segment cells count=222 2023-04-16 00:55:50,143 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=222, cellSize=10162, totalHeapSize=23328, min timestamp=1681606550125, max timestamp=1681606550140Num uniques 1122; 2023-04-16 00:55:50,143 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=51382, new segment data size=10162, suffix heap size=144656,new segment heap size=23328 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1122, new segment cells count=222 2023-04-16 00:55:50,143 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=222, cellSize=10162, totalHeapSize=23328, min timestamp=1681606550125, max timestamp=1681606550140Num uniques 1122; 2023-04-16 00:55:50,143 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=51382, new segment data size=10162, suffix heap size=144656,new segment heap size=23328 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1122, new segment cells count=222 2023-04-16 00:55:50,178 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=331, cellSize=15156, totalHeapSize=34664, min timestamp=1681606550163, max timestamp=1681606550175Num uniques 1081; 2023-04-16 00:55:50,178 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=49506, new segment data size=15156, suffix heap size=140392,new segment heap size=34664 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1081, new segment cells count=331 2023-04-16 00:55:50,178 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=331, cellSize=15156, totalHeapSize=34664, min timestamp=1681606550163, max timestamp=1681606550175Num uniques 1081; 2023-04-16 00:55:50,178 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=49506, new segment data size=15156, suffix heap size=140392,new segment heap size=34664 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1081, new segment cells count=331 2023-04-16 00:55:50,178 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=181, cellSize=8286, totalHeapSize=19064, min timestamp=1681606550163, max timestamp=1681606550175Num uniques 1081; 2023-04-16 00:55:50,178 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=49506, new segment data size=8286, suffix heap size=140392,new segment heap size=19064 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1081, new segment cells count=181 2023-04-16 00:55:50,183 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=245 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp/C/3d56fd5c4525452e988fc2410ca6af09 2023-04-16 00:55:50,187 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp/A/b627f7424e304af0a35e600740b3395d as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/A/b627f7424e304af0a35e600740b3395d 2023-04-16 00:55:50,190 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/A/b627f7424e304af0a35e600740b3395d, entries=150, sequenceid=245, filesize=11.8 K 2023-04-16 00:55:50,191 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp/B/ea3738a001ff4d7494adfc7f72a3b985 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/B/ea3738a001ff4d7494adfc7f72a3b985 2023-04-16 00:55:50,194 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/B/ea3738a001ff4d7494adfc7f72a3b985, entries=100, sequenceid=245, filesize=9.4 K 2023-04-16 00:55:50,195 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp/C/3d56fd5c4525452e988fc2410ca6af09 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/C/3d56fd5c4525452e988fc2410ca6af09 2023-04-16 00:55:50,207 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/C/3d56fd5c4525452e988fc2410ca6af09, entries=100, sequenceid=245, filesize=9.4 K 2023-04-16 00:55:50,208 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~20.13 KB/20610, heapSize ~60.47 KB/61920, currentSize=114.05 KB/116790 for 0abd14932d4ec4e5fdc6177f2c7cc55c in 451ms, sequenceid=245, compaction requested=true 2023-04-16 00:55:50,208 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 0abd14932d4ec4e5fdc6177f2c7cc55c: 2023-04-16 00:55:50,208 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. 2023-04-16 00:55:50,208 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:55:50,208 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:55:50,208 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:50,208 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:55:50,208 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:55:50,210 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:50,210 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:50,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:50,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:50,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:50,210 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:50,210 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:55:50,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:50,211 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(9158): Flush requested on 0abd14932d4ec4e5fdc6177f2c7cc55c 2023-04-16 00:55:50,211 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 0abd14932d4ec4e5fdc6177f2c7cc55c 3/3 column families, dataSize=120.76 KB heapSize=333.70 KB 2023-04-16 00:55:50,212 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0abd14932d4ec4e5fdc6177f2c7cc55c, store=A 2023-04-16 00:55:50,212 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 00:55:50,212 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0abd14932d4ec4e5fdc6177f2c7cc55c, store=B 2023-04-16 00:55:50,212 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 00:55:50,212 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0abd14932d4ec4e5fdc6177f2c7cc55c, store=C 2023-04-16 00:55:50,212 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 00:55:50,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:50,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:50,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:50,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:50,214 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:55:50,214 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@687b049e[Count = 0] remaining members to acquire global barrier 2023-04-16 00:55:50,214 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:55:50,214 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:50,217 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:50,217 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:55:50,217 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:50,217 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:50,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:50,217 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:55:50,217 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:55:50,217 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:55:50,218 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:55:50,218 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:50,219 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:55:50,219 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:50,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:50,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:50,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:50,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:50,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:50,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:50,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:50,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:50,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:50,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:50,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:55:50,221 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:55:50,221 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:55:50,221 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:55:50,221 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:55:50,221 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:55:50,221 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:55:50,223 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:50,223 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:50,223 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:50,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:50,223 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:50,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:50,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:50,224 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:50,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:50,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:50,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:50,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:50,225 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:50,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:50,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:50,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:50,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:50,226 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:50,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:50,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:50,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:50,228 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:50,228 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:50,228 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:50,228 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:55:50,228 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:50,228 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:55:50,228 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:50,228 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:50,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:50,228 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:50,228 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:55:50,228 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:50,228 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:50,228 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:50,228 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:50,228 DEBUG [Thread-3183] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:55:50,228 DEBUG [Thread-3183] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:55:50,228 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:50,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:50,232 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:50,232 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57254 deadline: 1681606610224, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:50,239 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:50,239 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57212 deadline: 1681606610228, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:50,245 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:50,245 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57202 deadline: 1681606610232, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:50,245 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:50,245 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57268 deadline: 1681606610232, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:50,329 DEBUG [Thread-3183] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:55:50,329 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:55:50,331 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:55:50,332 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:55:50,332 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:55:50,332 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:55:50,332 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:50,332 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:55:50,332 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:55:50,332 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:50,333 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:50,334 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:50,334 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:50,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:50,334 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:50,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:50,334 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:50,334 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:55:50,334 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:50,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:55:50,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:50,335 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:55:50,335 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:55:50,335 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:55:50,335 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:55:50,335 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:50,335 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:55:50,335 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:55:50,336 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:55:50,336 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. 2023-04-16 00:55:50,336 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. started... 2023-04-16 00:55:50,336 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. as already flushing 2023-04-16 00:55:50,336 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. 2023-04-16 00:55:50,336 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:55:50,336 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:55:50,336 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:50,336 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:55:50,336 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:55:50,336 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:50,336 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57254 deadline: 1681606610333, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:50,337 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:50,337 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:50,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:50,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:50,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:50,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:50,338 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:50,338 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:55:50,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:50,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:50,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:50,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:50,339 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:55:50,339 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@9d88023[Count = 0] remaining members to acquire global barrier 2023-04-16 00:55:50,339 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:55:50,339 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:50,340 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:50,340 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:50,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:50,341 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:55:50,341 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:55:50,341 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:55:50,341 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:50,341 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:55:50,344 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:55:50,344 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:50,344 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:50,344 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:50,345 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:55:50,345 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57212 deadline: 1681606610340, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:50,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:50,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:50,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:50,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:50,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:50,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:50,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:50,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:50,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:50,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:50,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:55:50,347 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:55:50,347 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:55:50,347 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:55:50,347 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:55:50,347 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:55:50,347 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:55:50,349 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:50,349 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:50,349 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:50,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:50,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:50,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:50,349 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:50,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:50,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:50,349 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:50,349 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:50,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:50,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:50,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:50,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:50,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:50,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:50,351 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:50,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:50,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:50,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:50,353 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:50,353 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:50,353 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:50,353 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:50,353 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:55:50,353 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:55:50,353 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:50,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:50,353 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:50,353 DEBUG [Thread-3183] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:55:50,353 DEBUG [Thread-3183] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:55:50,353 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:50,353 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:50,353 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:50,353 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:50,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:50,353 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:50,354 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:50,358 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:50,358 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57202 deadline: 1681606610346, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:50,358 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:50,358 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57268 deadline: 1681606610346, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:50,453 DEBUG [Thread-3183] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:55:50,454 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:55:50,456 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:55:50,457 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:55:50,457 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:55:50,457 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:55:50,457 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:50,457 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:55:50,457 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:55:50,457 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:50,457 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:50,459 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:50,459 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:50,459 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:50,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:50,459 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:50,459 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:55:50,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:50,459 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:50,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:55:50,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:50,460 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:55:50,460 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:55:50,460 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:55:50,460 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:55:50,460 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:50,460 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:55:50,461 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:55:50,461 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:55:50,461 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. 2023-04-16 00:55:50,461 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. started... 2023-04-16 00:55:50,461 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. as already flushing 2023-04-16 00:55:50,461 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. 2023-04-16 00:55:50,461 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:55:50,461 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:55:50,461 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:50,461 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:55:50,461 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:55:50,463 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:50,463 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:50,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:50,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:50,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:50,463 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:50,464 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:55:50,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:50,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:50,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:50,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:50,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:50,465 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:55:50,465 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4bc1dee3[Count = 0] remaining members to acquire global barrier 2023-04-16 00:55:50,465 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:55:50,465 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:50,466 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:50,466 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:50,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:50,466 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:55:50,466 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:55:50,466 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:55:50,466 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:50,467 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:55:50,468 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:55:50,468 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:50,468 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:50,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:50,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:50,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:50,468 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:55:50,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:50,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:50,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:50,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:50,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:50,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:50,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:50,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:55:50,470 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:55:50,470 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:55:50,470 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:55:50,470 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:55:50,470 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:55:50,470 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:55:50,473 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:50,473 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:50,473 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:50,473 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:50,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:50,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:50,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:50,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:50,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:50,474 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:50,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:50,474 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:50,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:50,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:50,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:50,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:50,475 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:50,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:50,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:50,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:50,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:50,479 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:50,479 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:50,479 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:50,479 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:50,479 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:50,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:50,479 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:55:50,479 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:55:50,479 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:50,479 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:50,479 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:50,479 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:50,479 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:50,479 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:50,479 DEBUG [Thread-3183] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:55:50,479 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:55:50,479 DEBUG [Thread-3183] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:55:50,480 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:50,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:50,542 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:50,542 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57254 deadline: 1681606610537, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:50,551 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:50,551 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57212 deadline: 1681606610547, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:50,563 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:50,563 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57202 deadline: 1681606610559, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:50,565 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:50,565 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57268 deadline: 1681606610560, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:50,580 DEBUG [Thread-3183] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:55:50,580 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:55:50,582 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:55:50,583 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:55:50,583 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:55:50,583 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:55:50,583 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:50,583 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:55:50,583 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:55:50,584 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:50,584 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:50,585 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:50,585 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:50,585 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:50,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:50,585 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:50,585 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:55:50,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:50,586 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:50,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:55:50,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:50,586 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:55:50,586 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:55:50,586 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:55:50,586 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:55:50,586 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:50,586 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:55:50,587 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:55:50,587 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:55:50,587 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. 2023-04-16 00:55:50,587 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. started... 2023-04-16 00:55:50,587 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. as already flushing 2023-04-16 00:55:50,587 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. 2023-04-16 00:55:50,587 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:55:50,587 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:55:50,587 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:50,588 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:55:50,588 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:55:50,589 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:50,589 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:50,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:50,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:50,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:50,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:50,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:50,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:50,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:50,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:50,591 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:55:50,591 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5b1f4029[Count = 0] remaining members to acquire global barrier 2023-04-16 00:55:50,591 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:50,591 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:55:50,591 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:55:50,591 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:50,593 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:50,593 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:50,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:50,593 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:55:50,593 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:55:50,593 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:55:50,593 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:50,593 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:55:50,594 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:50,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:50,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:50,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:50,594 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:55:50,595 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:50,595 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:55:50,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:50,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:50,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:50,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:50,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:50,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:50,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:50,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:55:50,597 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:55:50,597 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:55:50,597 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:55:50,597 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:55:50,597 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:55:50,597 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:55:50,598 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:50,598 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:50,599 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:50,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:50,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:50,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:50,599 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:50,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:50,599 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:50,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:50,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:50,599 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:50,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:50,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:50,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:50,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:50,601 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:50,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:50,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:50,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:50,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:50,602 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:50,602 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:50,602 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:50,602 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:50,602 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:55:50,603 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:55:50,602 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:50,603 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:50,603 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:50,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:50,603 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:50,603 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:50,603 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:50,603 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:55:50,603 DEBUG [Thread-3183] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:55:50,603 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:50,603 DEBUG [Thread-3183] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:55:50,603 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:50,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:50,621 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=418 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp/A/99f7e63612134d64b8ccc795b3969ad8 2023-04-16 00:55:50,631 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=418 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp/B/85223b01a4c54d7ca4a5d37ef2deb5ad 2023-04-16 00:55:50,643 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=418 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp/C/b253421c9ea04af0933667f51d1d8fa0 2023-04-16 00:55:50,646 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp/A/99f7e63612134d64b8ccc795b3969ad8 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/A/99f7e63612134d64b8ccc795b3969ad8 2023-04-16 00:55:50,650 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/A/99f7e63612134d64b8ccc795b3969ad8, entries=200, sequenceid=418, filesize=14.3 K 2023-04-16 00:55:50,650 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp/B/85223b01a4c54d7ca4a5d37ef2deb5ad as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/B/85223b01a4c54d7ca4a5d37ef2deb5ad 2023-04-16 00:55:50,654 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/B/85223b01a4c54d7ca4a5d37ef2deb5ad, entries=150, sequenceid=418, filesize=11.9 K 2023-04-16 00:55:50,655 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp/C/b253421c9ea04af0933667f51d1d8fa0 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/C/b253421c9ea04af0933667f51d1d8fa0 2023-04-16 00:55:50,659 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/C/b253421c9ea04af0933667f51d1d8fa0, entries=150, sequenceid=418, filesize=11.9 K 2023-04-16 00:55:50,660 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~120.76 KB/123660, heapSize ~333.66 KB/341664, currentSize=60.38 KB/61830 for 0abd14932d4ec4e5fdc6177f2c7cc55c in 449ms, sequenceid=418, compaction requested=true 2023-04-16 00:55:50,660 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 0abd14932d4ec4e5fdc6177f2c7cc55c: 2023-04-16 00:55:50,660 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 00:55:50,660 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 00:55:50,660 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-16 00:55:50,660 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-16 00:55:50,660 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-16 00:55:50,662 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 46344 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-16 00:55:50,662 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HStore(1912): 0abd14932d4ec4e5fdc6177f2c7cc55c/B is initiating minor compaction (all files) 2023-04-16 00:55:50,662 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 0abd14932d4ec4e5fdc6177f2c7cc55c/B in TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. 2023-04-16 00:55:50,662 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/B/544c2a6ab4d04325accff040f2e4827d, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/B/4205e1d340c241bfb527f56828df10e3, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/B/ea3738a001ff4d7494adfc7f72a3b985, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/B/85223b01a4c54d7ca4a5d37ef2deb5ad] into tmpdir=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp, totalSize=45.3 K 2023-04-16 00:55:50,663 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 51174 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-16 00:55:50,663 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting 544c2a6ab4d04325accff040f2e4827d, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=204, earliestPutTs=1681606549028 2023-04-16 00:55:50,663 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HStore(1912): 0abd14932d4ec4e5fdc6177f2c7cc55c/A is initiating minor compaction (all files) 2023-04-16 00:55:50,663 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 0abd14932d4ec4e5fdc6177f2c7cc55c/A in TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. 2023-04-16 00:55:50,663 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/A/cba347c854c54e6091c926834313b1c9, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/A/e8351b101df84b41b3666061b9a9d395, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/A/b627f7424e304af0a35e600740b3395d, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/A/99f7e63612134d64b8ccc795b3969ad8] into tmpdir=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp, totalSize=50.0 K 2023-04-16 00:55:50,664 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting 4205e1d340c241bfb527f56828df10e3, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=239, earliestPutTs=1681606549124 2023-04-16 00:55:50,666 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.Compactor(207): Compacting cba347c854c54e6091c926834313b1c9, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=204, earliestPutTs=1681606549028 2023-04-16 00:55:50,666 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting ea3738a001ff4d7494adfc7f72a3b985, keycount=100, bloomtype=ROW, size=9.4 K, encoding=NONE, compression=NONE, seqNum=245, earliestPutTs=1681606549752 2023-04-16 00:55:50,667 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.Compactor(207): Compacting e8351b101df84b41b3666061b9a9d395, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=239, earliestPutTs=1681606549124 2023-04-16 00:55:50,667 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting 85223b01a4c54d7ca4a5d37ef2deb5ad, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=418, earliestPutTs=1681606550190 2023-04-16 00:55:50,667 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.Compactor(207): Compacting b627f7424e304af0a35e600740b3395d, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=245, earliestPutTs=1681606549747 2023-04-16 00:55:50,667 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.Compactor(207): Compacting 99f7e63612134d64b8ccc795b3969ad8, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=418, earliestPutTs=1681606550190 2023-04-16 00:55:50,677 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] throttle.PressureAwareThroughputController(145): 0abd14932d4ec4e5fdc6177f2c7cc55c#B#compaction#456 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-16 00:55:50,678 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] throttle.PressureAwareThroughputController(145): 0abd14932d4ec4e5fdc6177f2c7cc55c#A#compaction#457 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-16 00:55:50,694 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp/B/980a4fbe8fc041c4816337e40a02c1b1 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/B/980a4fbe8fc041c4816337e40a02c1b1 2023-04-16 00:55:50,701 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 0abd14932d4ec4e5fdc6177f2c7cc55c/B of 0abd14932d4ec4e5fdc6177f2c7cc55c into 980a4fbe8fc041c4816337e40a02c1b1(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 00:55:50,701 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0abd14932d4ec4e5fdc6177f2c7cc55c: 2023-04-16 00:55:50,701 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c., storeName=0abd14932d4ec4e5fdc6177f2c7cc55c/B, priority=12, startTime=1681606550660; duration=0sec 2023-04-16 00:55:50,701 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 00:55:50,701 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-16 00:55:50,702 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 34342 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-16 00:55:50,702 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HStore(1912): 0abd14932d4ec4e5fdc6177f2c7cc55c/C is initiating minor compaction (all files) 2023-04-16 00:55:50,702 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 0abd14932d4ec4e5fdc6177f2c7cc55c/C in TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. 2023-04-16 00:55:50,702 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/C/14da418ff71c41c69a3febacebd88a8b, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/C/3d56fd5c4525452e988fc2410ca6af09, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/C/b253421c9ea04af0933667f51d1d8fa0] into tmpdir=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp, totalSize=33.5 K 2023-04-16 00:55:50,703 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting 14da418ff71c41c69a3febacebd88a8b, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=239, earliestPutTs=1681606549124 2023-04-16 00:55:50,703 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting 3d56fd5c4525452e988fc2410ca6af09, keycount=100, bloomtype=ROW, size=9.4 K, encoding=NONE, compression=NONE, seqNum=245, earliestPutTs=1681606549752 2023-04-16 00:55:50,703 DEBUG [Thread-3183] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:55:50,704 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting b253421c9ea04af0933667f51d1d8fa0, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=418, earliestPutTs=1681606550190 2023-04-16 00:55:50,704 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:55:50,707 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:55:50,708 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:55:50,708 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:55:50,708 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:55:50,708 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:50,708 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:55:50,708 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:55:50,708 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:50,708 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:50,709 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] throttle.PressureAwareThroughputController(145): 0abd14932d4ec4e5fdc6177f2c7cc55c#C#compaction#458 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-16 00:55:50,710 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:50,710 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:50,711 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:50,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:50,711 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:50,711 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:55:50,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:50,712 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:50,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:55:50,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:50,713 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:55:50,713 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:55:50,713 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:55:50,713 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:55:50,713 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:50,713 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:55:50,714 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:55:50,714 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:55:50,714 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. 2023-04-16 00:55:50,715 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. started... 2023-04-16 00:55:50,715 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 0abd14932d4ec4e5fdc6177f2c7cc55c 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-16 00:55:50,716 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0abd14932d4ec4e5fdc6177f2c7cc55c, store=A 2023-04-16 00:55:50,716 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:55:50,716 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0abd14932d4ec4e5fdc6177f2c7cc55c, store=B 2023-04-16 00:55:50,717 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:55:50,717 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0abd14932d4ec4e5fdc6177f2c7cc55c, store=C 2023-04-16 00:55:50,717 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:55:50,720 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp/C/52e41293bba9416cbf56a974c13d0db5 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/C/52e41293bba9416cbf56a974c13d0db5 2023-04-16 00:55:50,725 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=431 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp/A/af6f6d3322aa401a941aaed00e280519 2023-04-16 00:55:50,726 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 0abd14932d4ec4e5fdc6177f2c7cc55c/C of 0abd14932d4ec4e5fdc6177f2c7cc55c into 52e41293bba9416cbf56a974c13d0db5(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 00:55:50,726 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0abd14932d4ec4e5fdc6177f2c7cc55c: 2023-04-16 00:55:50,726 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c., storeName=0abd14932d4ec4e5fdc6177f2c7cc55c/C, priority=13, startTime=1681606550660; duration=0sec 2023-04-16 00:55:50,726 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 00:55:50,741 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=431 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp/B/df6dc2d800084e53bebca094f7dfe523 2023-04-16 00:55:50,751 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=431 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp/C/a807a8475d49448ea8dba33fc8ab598b 2023-04-16 00:55:50,755 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp/A/af6f6d3322aa401a941aaed00e280519 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/A/af6f6d3322aa401a941aaed00e280519 2023-04-16 00:55:50,758 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/A/af6f6d3322aa401a941aaed00e280519, entries=150, sequenceid=431, filesize=11.9 K 2023-04-16 00:55:50,759 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp/B/df6dc2d800084e53bebca094f7dfe523 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/B/df6dc2d800084e53bebca094f7dfe523 2023-04-16 00:55:50,763 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/B/df6dc2d800084e53bebca094f7dfe523, entries=150, sequenceid=431, filesize=11.9 K 2023-04-16 00:55:50,764 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp/C/a807a8475d49448ea8dba33fc8ab598b as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/C/a807a8475d49448ea8dba33fc8ab598b 2023-04-16 00:55:50,770 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/C/a807a8475d49448ea8dba33fc8ab598b, entries=150, sequenceid=431, filesize=11.9 K 2023-04-16 00:55:50,771 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=0 B/0 for 0abd14932d4ec4e5fdc6177f2c7cc55c in 56ms, sequenceid=431, compaction requested=false 2023-04-16 00:55:50,771 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 0abd14932d4ec4e5fdc6177f2c7cc55c: 2023-04-16 00:55:50,771 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. 2023-04-16 00:55:50,771 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:55:50,771 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:55:50,771 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:50,771 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:55:50,771 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:55:50,773 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:50,773 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:50,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:50,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:50,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:50,773 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:50,773 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:55:50,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:50,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:50,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:50,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:50,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:50,776 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:55:50,776 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6bf0e020[Count = 0] remaining members to acquire global barrier 2023-04-16 00:55:50,776 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:55:50,776 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:50,777 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:50,778 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:50,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:50,778 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:55:50,778 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:55:50,778 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:55:50,779 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:55:50,779 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:50,779 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:50,779 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:55:50,779 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:55:50,779 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:55:50,779 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:55:50,780 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:55:50,780 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:55:50,780 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:55:50,780 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:55:50,783 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:50,783 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:50,783 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:50,783 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:50,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:50,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:50,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:50,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:50,783 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:50,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:50,783 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:50,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:50,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:50,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:50,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:50,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:50,785 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:50,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:50,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:50,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:50,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:50,787 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:50,787 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:50,787 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:50,787 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:50,787 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:50,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:50,787 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:55:50,787 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:55:50,787 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:50,787 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:50,787 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:50,787 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:50,787 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:50,787 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:50,787 DEBUG [Thread-3183] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:55:50,787 DEBUG [Thread-3183] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:55:50,788 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:50,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:50,788 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:55:50,861 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(9158): Flush requested on 0abd14932d4ec4e5fdc6177f2c7cc55c 2023-04-16 00:55:50,861 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 0abd14932d4ec4e5fdc6177f2c7cc55c 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-16 00:55:50,862 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0abd14932d4ec4e5fdc6177f2c7cc55c, store=A 2023-04-16 00:55:50,862 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:55:50,862 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0abd14932d4ec4e5fdc6177f2c7cc55c, store=B 2023-04-16 00:55:50,862 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:55:50,862 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0abd14932d4ec4e5fdc6177f2c7cc55c, store=C 2023-04-16 00:55:50,862 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:55:50,888 DEBUG [Thread-3183] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:55:50,888 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:55:50,890 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:55:50,893 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:55:50,893 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:55:50,893 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:55:50,893 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:50,893 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:55:50,893 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:55:50,893 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:50,893 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:50,894 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:50,894 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:50,894 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:50,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:50,895 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:50,895 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:55:50,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:50,895 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:50,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:55:50,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:50,895 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:55:50,896 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:55:50,896 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:55:50,896 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:55:50,896 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:50,896 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:55:50,896 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:55:50,896 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:55:50,896 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. 2023-04-16 00:55:50,896 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. started... 2023-04-16 00:55:50,896 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. as already flushing 2023-04-16 00:55:50,896 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. 2023-04-16 00:55:50,896 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:55:50,896 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:55:50,896 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:50,897 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:55:50,897 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:55:50,898 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:50,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:50,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:50,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:50,898 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:50,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:50,898 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:50,898 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:55:50,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:50,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:50,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:50,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:50,899 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:55:50,899 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@26e4bce5[Count = 0] remaining members to acquire global barrier 2023-04-16 00:55:50,899 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:55:50,900 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:50,901 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:50,901 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:50,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:50,901 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:55:50,901 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:55:50,901 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:55:50,901 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:50,902 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:55:50,903 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:50,903 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:55:50,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:50,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:50,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:50,903 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:50,903 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:55:50,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:50,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:50,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:50,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:50,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:50,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:50,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:50,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:55:50,905 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:55:50,905 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:55:50,905 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:55:50,905 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:55:50,905 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:55:50,905 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:55:50,908 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:50,908 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:50,908 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:50,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:50,908 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:50,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:50,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:50,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:50,909 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:50,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:50,909 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:50,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:50,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:50,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:50,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:50,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:50,910 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:50,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:50,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:50,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:50,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:50,913 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:50,913 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:50,913 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:50,913 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:50,913 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:50,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:50,913 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:55:50,913 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:55:50,913 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:50,913 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:50,913 DEBUG [Thread-3183] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:55:50,913 DEBUG [Thread-3183] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:55:50,913 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:50,913 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:50,913 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:50,913 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:50,913 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:50,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:50,927 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:50,927 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57202 deadline: 1681606610920, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:50,928 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681606550889, max timestamp=1681606550926Num uniques 859; 2023-04-16 00:55:50,928 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-16 00:55:50,928 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681606550889, max timestamp=1681606550926Num uniques 859; 2023-04-16 00:55:50,928 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681606550889, max timestamp=1681606550926Num uniques 859; 2023-04-16 00:55:50,928 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-16 00:55:50,928 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-16 00:55:50,963 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:50,963 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57268 deadline: 1681606610960, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:50,967 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:50,967 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57254 deadline: 1681606610961, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:50,974 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:50,975 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 166 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57212 deadline: 1681606610962, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:51,013 DEBUG [Thread-3183] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:55:51,014 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:55:51,015 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:55:51,016 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:55:51,016 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:55:51,016 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:55:51,016 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:51,016 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:55:51,017 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:55:51,017 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:51,017 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:51,018 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:51,018 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:51,018 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:51,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:51,018 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:51,018 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:55:51,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:51,019 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:51,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:55:51,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:51,019 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:55:51,019 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:55:51,020 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:55:51,020 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:55:51,020 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:51,020 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:55:51,020 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:55:51,020 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:55:51,020 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. 2023-04-16 00:55:51,020 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. started... 2023-04-16 00:55:51,020 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. as already flushing 2023-04-16 00:55:51,020 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. 2023-04-16 00:55:51,020 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:55:51,020 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:55:51,020 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:51,020 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:55:51,020 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:55:51,021 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:51,021 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:51,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:51,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:51,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:51,022 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:51,022 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:55:51,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:51,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:51,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:51,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:51,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:51,023 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:55:51,023 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6317481c[Count = 0] remaining members to acquire global barrier 2023-04-16 00:55:51,023 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:55:51,023 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:51,024 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:51,024 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:51,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:51,024 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:51,024 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:55:51,024 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:55:51,024 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:55:51,024 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:55:51,026 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:55:51,027 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:51,027 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:51,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:51,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:51,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:51,027 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:55:51,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:51,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:51,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:51,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:51,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:51,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:51,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:51,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:55:51,039 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:55:51,039 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:55:51,039 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:55:51,039 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:55:51,040 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:55:51,040 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:55:51,040 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:51,040 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57202 deadline: 1681606611038, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:51,041 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:51,041 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:51,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:51,041 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:51,041 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:51,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:51,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:51,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:51,042 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:51,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:51,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:51,042 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:51,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:51,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:51,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:51,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:51,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:51,043 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:51,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:51,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:51,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:51,045 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:51,045 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:51,045 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:51,045 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:51,045 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:51,045 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:51,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:51,045 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:55:51,045 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:55:51,045 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:51,045 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:51,046 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:51,046 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:51,046 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:51,046 DEBUG [Thread-3183] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:55:51,046 DEBUG [Thread-3183] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:55:51,046 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:51,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:51,069 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:51,069 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57268 deadline: 1681606611064, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:51,076 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:51,076 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57254 deadline: 1681606611068, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:51,080 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:51,080 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 168 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57212 deadline: 1681606611075, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:51,097 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp/A/9aca28d6a62f4703a365acdbc5622b5e as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/A/9aca28d6a62f4703a365acdbc5622b5e 2023-04-16 00:55:51,101 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 0abd14932d4ec4e5fdc6177f2c7cc55c/A of 0abd14932d4ec4e5fdc6177f2c7cc55c into 9aca28d6a62f4703a365acdbc5622b5e(size=12.5 K), total size for store is 24.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 00:55:51,101 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0abd14932d4ec4e5fdc6177f2c7cc55c: 2023-04-16 00:55:51,101 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c., storeName=0abd14932d4ec4e5fdc6177f2c7cc55c/A, priority=12, startTime=1681606550660; duration=0sec 2023-04-16 00:55:51,101 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 00:55:51,146 DEBUG [Thread-3183] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:55:51,146 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:55:51,148 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:55:51,150 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:55:51,150 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:55:51,150 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:55:51,150 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:51,150 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:55:51,150 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:55:51,151 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:51,151 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:51,152 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:51,152 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:51,152 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:51,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:51,152 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:51,152 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:55:51,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:51,153 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:51,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:55:51,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:51,155 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:55:51,155 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:55:51,155 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:55:51,155 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:51,155 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:55:51,155 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:55:51,155 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:55:51,155 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. 2023-04-16 00:55:51,155 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. started... 2023-04-16 00:55:51,155 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. as already flushing 2023-04-16 00:55:51,155 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. 2023-04-16 00:55:51,155 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:55:51,156 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:55:51,156 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:51,156 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:55:51,156 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:55:51,159 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:51,159 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:51,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:51,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:51,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:51,160 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:51,160 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:55:51,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:51,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:51,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:51,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:51,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:51,161 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:55:51,161 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@42a8425f[Count = 0] remaining members to acquire global barrier 2023-04-16 00:55:51,161 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:55:51,161 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:51,162 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:51,162 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:51,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:51,162 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:55:51,163 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:55:51,163 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:55:51,163 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:51,163 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:55:51,164 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:55:51,164 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:51,164 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:51,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:51,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:51,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:51,164 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:55:51,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:51,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:51,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:51,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:51,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:51,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:51,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:51,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:55:51,167 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:55:51,167 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:55:51,167 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:55:51,167 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:55:51,167 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:55:51,167 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:55:51,168 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:51,169 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:51,169 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:51,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:51,169 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:51,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:51,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:51,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:51,169 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:51,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:51,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:51,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:51,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:51,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:51,170 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:51,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:51,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:51,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:51,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:51,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:51,172 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:51,175 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:51,175 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:51,175 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:51,175 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:51,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:51,175 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:55:51,175 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:55:51,175 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:51,176 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:51,176 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:51,176 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:51,176 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:51,176 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:51,176 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:51,176 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:51,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:51,176 DEBUG [Thread-3183] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:55:51,176 DEBUG [Thread-3183] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:55:51,244 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:51,244 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57202 deadline: 1681606611242, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:51,273 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=443 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp/A/4653c4bed25c4a069797b7ac706e310b 2023-04-16 00:55:51,274 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:51,274 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57268 deadline: 1681606611271, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:51,276 DEBUG [Thread-3183] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:55:51,277 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:55:51,278 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:55:51,279 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:55:51,279 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:55:51,279 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:55:51,279 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:51,280 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:55:51,280 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:55:51,280 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:51,280 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:51,282 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:51,282 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:51,282 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:51,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:51,282 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:51,282 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:55:51,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:51,282 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:51,283 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57254 deadline: 1681606611278, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:51,283 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:51,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:55:51,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:51,283 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:55:51,284 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:55:51,284 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:55:51,284 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:55:51,284 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:51,284 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:55:51,288 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:55:51,288 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:55:51,288 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. 2023-04-16 00:55:51,288 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. started... 2023-04-16 00:55:51,288 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. as already flushing 2023-04-16 00:55:51,288 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. 2023-04-16 00:55:51,288 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:51,288 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:55:51,289 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:55:51,289 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:51,288 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 170 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57212 deadline: 1681606611282, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:51,289 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:55:51,289 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:55:51,290 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:51,290 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:51,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:51,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:51,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:51,291 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:51,291 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:55:51,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:51,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:51,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:51,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:51,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:51,292 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:55:51,292 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2e355bca[Count = 0] remaining members to acquire global barrier 2023-04-16 00:55:51,292 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:55:51,292 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:51,293 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:51,293 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:51,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:51,294 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:55:51,294 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:55:51,294 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:55:51,294 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:51,294 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:55:51,296 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:51,296 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:55:51,296 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:51,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:51,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:51,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:51,296 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:55:51,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:51,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:51,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:51,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:51,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:51,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:51,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:51,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:55:51,300 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:55:51,300 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:55:51,300 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:55:51,300 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:55:51,300 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:55:51,300 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:55:51,301 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:51,301 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:51,301 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:51,301 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:51,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:51,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:51,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:51,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:51,302 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:51,302 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:51,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:51,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:51,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:51,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:51,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:51,303 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:51,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:51,311 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:51,311 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:51,311 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:51,311 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:51,311 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:51,311 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:51,311 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:51,311 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:51,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:51,311 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:55:51,311 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:55:51,312 DEBUG [Thread-3183] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:55:51,312 DEBUG [Thread-3183] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:55:51,312 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:51,312 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:51,312 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:51,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:51,312 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:55:51,312 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:51,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:51,412 DEBUG [Thread-3183] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:55:51,412 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:55:51,414 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:55:51,415 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:55:51,415 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:55:51,415 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:55:51,415 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:51,415 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:55:51,415 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:55:51,416 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:51,416 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:51,418 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:51,418 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:51,418 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:51,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:51,419 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:51,419 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:55:51,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:51,419 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:51,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:55:51,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:51,420 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:55:51,420 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:55:51,420 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:55:51,420 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:55:51,420 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:51,420 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:55:51,421 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:55:51,421 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:55:51,421 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. 2023-04-16 00:55:51,421 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. started... 2023-04-16 00:55:51,421 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. as already flushing 2023-04-16 00:55:51,421 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. 2023-04-16 00:55:51,421 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:55:51,421 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:55:51,421 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:51,421 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:55:51,421 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:55:51,422 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:51,422 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:51,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:51,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:51,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:51,423 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:51,423 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:55:51,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:51,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:51,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:51,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:51,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:51,424 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:55:51,424 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2d35c97d[Count = 0] remaining members to acquire global barrier 2023-04-16 00:55:51,424 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:55:51,424 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:51,425 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:51,425 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:51,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:51,426 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:51,426 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:55:51,426 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:55:51,426 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:55:51,426 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:55:51,427 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:55:51,427 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:51,427 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:51,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:51,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:51,427 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:55:51,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:51,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:51,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:51,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:51,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:51,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:51,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:51,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:51,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:55:51,431 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:55:51,431 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:55:51,431 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:55:51,431 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:55:51,431 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:55:51,431 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:55:51,433 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:51,433 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:51,433 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:51,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:51,434 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:51,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:51,434 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:51,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:51,437 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:51,439 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:51,439 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:51,439 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:51,439 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:51,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:51,439 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:55:51,439 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:55:51,440 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:51,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:51,440 DEBUG [Thread-3183] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:55:51,440 DEBUG [Thread-3183] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:55:51,444 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:51,444 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:51,444 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:51,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:51,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:51,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:51,444 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:51,444 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:51,444 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:51,444 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:51,444 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:51,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:51,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:51,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:51,445 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:55:51,476 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:51,476 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57228 deadline: 1681606611471, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:51,479 DEBUG [Thread-3177] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4165 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c., hostname=jenkins-hbase4.apache.org,34975,1681606419217, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor32.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-16 00:55:51,540 DEBUG [Thread-3183] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:55:51,540 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:55:51,542 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:55:51,543 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:55:51,543 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:55:51,543 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:55:51,543 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:51,543 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:55:51,543 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:55:51,543 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:51,543 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:51,544 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:51,545 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:51,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:51,545 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:51,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:51,545 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:51,545 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:51,545 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:55:51,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:55:51,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:51,546 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:55:51,546 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:55:51,546 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:55:51,546 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:55:51,546 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:51,546 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:55:51,546 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:55:51,546 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:55:51,546 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. 2023-04-16 00:55:51,546 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. started... 2023-04-16 00:55:51,546 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. as already flushing 2023-04-16 00:55:51,546 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. 2023-04-16 00:55:51,547 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:55:51,547 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:55:51,547 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:51,547 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:55:51,547 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:55:51,549 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:51,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:51,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:51,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:51,549 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:51,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:51,549 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:51,549 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:55:51,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:51,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:51,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:51,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:51,550 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:55:51,550 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@743b336f[Count = 0] remaining members to acquire global barrier 2023-04-16 00:55:51,550 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:55:51,550 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:51,552 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:51,552 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:51,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:51,552 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:55:51,552 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:55:51,552 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:55:51,553 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:51,553 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:55:51,554 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:51,554 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:55:51,554 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:51,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:51,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:51,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:51,554 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:55:51,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:51,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:51,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:51,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:51,555 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:51,555 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57202 deadline: 1681606611546, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:51,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:51,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:51,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:51,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:55:51,557 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:55:51,557 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:55:51,557 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:55:51,557 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:55:51,557 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:55:51,557 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:55:51,558 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:51,558 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:51,558 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:51,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:51,558 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:51,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:51,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:51,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:51,559 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:51,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:51,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:51,559 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:51,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:51,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:51,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:51,560 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:51,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:51,561 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:51,561 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:51,561 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:51,561 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:55:51,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:51,561 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:51,561 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:55:51,561 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:51,562 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:51,561 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:51,562 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:51,562 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:51,562 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:51,562 DEBUG [Thread-3183] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:55:51,562 DEBUG [Thread-3183] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:55:51,562 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:51,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:51,562 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:55:51,562 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:51,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:51,579 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:51,579 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 162 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57268 deadline: 1681606611577, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:51,587 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:51,587 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 159 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57254 deadline: 1681606611584, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:51,594 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:51,594 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 172 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57212 deadline: 1681606611591, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:51,662 DEBUG [Thread-3183] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:55:51,662 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:55:51,664 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:55:51,665 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:55:51,665 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:55:51,665 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:55:51,665 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:51,665 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:55:51,665 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:55:51,665 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:51,665 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:51,671 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:51,671 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:51,671 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:51,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:51,672 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:51,672 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:55:51,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:51,672 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:51,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:55:51,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:51,673 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:55:51,673 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:55:51,673 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:55:51,673 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:55:51,673 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:51,673 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:55:51,673 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:55:51,673 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:55:51,673 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. 2023-04-16 00:55:51,673 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. started... 2023-04-16 00:55:51,673 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. as already flushing 2023-04-16 00:55:51,673 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. 2023-04-16 00:55:51,673 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:55:51,673 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:55:51,674 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:51,674 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:55:51,674 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:55:51,675 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:51,675 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:51,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:51,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:51,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:51,675 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:51,675 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:55:51,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:51,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:51,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:51,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:51,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:51,676 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:55:51,676 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@74813d0d[Count = 0] remaining members to acquire global barrier 2023-04-16 00:55:51,676 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:55:51,676 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:51,678 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:51,679 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:51,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:51,679 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:55:51,679 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:55:51,679 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:55:51,680 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:55:51,680 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:51,680 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:55:51,684 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:51,688 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:55:51,688 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:55:51,688 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:55:51,688 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:55:51,688 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:55:51,688 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:55:51,688 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:55:51,690 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=443 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp/B/928f079b7c704dad9afde3a70a9c43f8 2023-04-16 00:55:51,690 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:51,690 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:51,690 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:51,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:51,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:51,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:51,691 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:51,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:51,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:51,691 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:51,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:51,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:51,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:51,692 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:51,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:51,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:51,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:51,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:51,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:51,693 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:51,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:51,703 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:51,703 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:51,703 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:51,703 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:55:51,703 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:51,703 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:51,703 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:51,703 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:51,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:51,704 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:51,703 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:55:51,704 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:51,704 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:51,704 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:51,704 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:51,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:51,704 DEBUG [Thread-3183] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:55:51,704 DEBUG [Thread-3183] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:55:51,711 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=443 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp/C/1f2310154add4781b73f380f5a9a8949 2023-04-16 00:55:51,715 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp/A/4653c4bed25c4a069797b7ac706e310b as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/A/4653c4bed25c4a069797b7ac706e310b 2023-04-16 00:55:51,721 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/A/4653c4bed25c4a069797b7ac706e310b, entries=200, sequenceid=443, filesize=14.3 K 2023-04-16 00:55:51,721 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp/B/928f079b7c704dad9afde3a70a9c43f8 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/B/928f079b7c704dad9afde3a70a9c43f8 2023-04-16 00:55:51,724 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/B/928f079b7c704dad9afde3a70a9c43f8, entries=150, sequenceid=443, filesize=11.9 K 2023-04-16 00:55:51,725 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp/C/1f2310154add4781b73f380f5a9a8949 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/C/1f2310154add4781b73f380f5a9a8949 2023-04-16 00:55:51,728 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/C/1f2310154add4781b73f380f5a9a8949, entries=150, sequenceid=443, filesize=11.9 K 2023-04-16 00:55:51,729 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 0abd14932d4ec4e5fdc6177f2c7cc55c in 868ms, sequenceid=443, compaction requested=true 2023-04-16 00:55:51,729 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 0abd14932d4ec4e5fdc6177f2c7cc55c: 2023-04-16 00:55:51,729 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-16 00:55:51,729 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 00:55:51,729 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-16 00:55:51,730 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 00:55:51,730 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 00:55:51,732 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37138 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-16 00:55:51,732 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 39578 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-16 00:55:51,732 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HStore(1912): 0abd14932d4ec4e5fdc6177f2c7cc55c/B is initiating minor compaction (all files) 2023-04-16 00:55:51,732 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HStore(1912): 0abd14932d4ec4e5fdc6177f2c7cc55c/A is initiating minor compaction (all files) 2023-04-16 00:55:51,733 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 0abd14932d4ec4e5fdc6177f2c7cc55c/B in TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. 2023-04-16 00:55:51,733 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 0abd14932d4ec4e5fdc6177f2c7cc55c/A in TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. 2023-04-16 00:55:51,733 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/B/980a4fbe8fc041c4816337e40a02c1b1, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/B/df6dc2d800084e53bebca094f7dfe523, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/B/928f079b7c704dad9afde3a70a9c43f8] into tmpdir=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp, totalSize=36.3 K 2023-04-16 00:55:51,733 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/A/9aca28d6a62f4703a365acdbc5622b5e, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/A/af6f6d3322aa401a941aaed00e280519, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/A/4653c4bed25c4a069797b7ac706e310b] into tmpdir=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp, totalSize=38.7 K 2023-04-16 00:55:51,733 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting 980a4fbe8fc041c4816337e40a02c1b1, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=418, earliestPutTs=1681606550190 2023-04-16 00:55:51,733 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.Compactor(207): Compacting 9aca28d6a62f4703a365acdbc5622b5e, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=418, earliestPutTs=1681606550190 2023-04-16 00:55:51,734 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting df6dc2d800084e53bebca094f7dfe523, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=431, earliestPutTs=1681606550222 2023-04-16 00:55:51,734 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.Compactor(207): Compacting af6f6d3322aa401a941aaed00e280519, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=431, earliestPutTs=1681606550222 2023-04-16 00:55:51,734 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting 928f079b7c704dad9afde3a70a9c43f8, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=443, earliestPutTs=1681606550858 2023-04-16 00:55:51,734 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.Compactor(207): Compacting 4653c4bed25c4a069797b7ac706e310b, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=443, earliestPutTs=1681606550856 2023-04-16 00:55:51,749 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] throttle.PressureAwareThroughputController(145): 0abd14932d4ec4e5fdc6177f2c7cc55c#B#compaction#465 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-16 00:55:51,750 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] throttle.PressureAwareThroughputController(145): 0abd14932d4ec4e5fdc6177f2c7cc55c#A#compaction#466 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-16 00:55:51,781 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp/B/bf5fcc1a45b44735aff89fb4837aec89 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/B/bf5fcc1a45b44735aff89fb4837aec89 2023-04-16 00:55:51,787 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 0abd14932d4ec4e5fdc6177f2c7cc55c/B of 0abd14932d4ec4e5fdc6177f2c7cc55c into bf5fcc1a45b44735aff89fb4837aec89(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 00:55:51,787 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0abd14932d4ec4e5fdc6177f2c7cc55c: 2023-04-16 00:55:51,787 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c., storeName=0abd14932d4ec4e5fdc6177f2c7cc55c/B, priority=13, startTime=1681606551729; duration=0sec 2023-04-16 00:55:51,787 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 00:55:51,787 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-16 00:55:51,788 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37138 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-16 00:55:51,788 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HStore(1912): 0abd14932d4ec4e5fdc6177f2c7cc55c/C is initiating minor compaction (all files) 2023-04-16 00:55:51,788 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 0abd14932d4ec4e5fdc6177f2c7cc55c/C in TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. 2023-04-16 00:55:51,789 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/C/52e41293bba9416cbf56a974c13d0db5, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/C/a807a8475d49448ea8dba33fc8ab598b, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/C/1f2310154add4781b73f380f5a9a8949] into tmpdir=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp, totalSize=36.3 K 2023-04-16 00:55:51,789 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting 52e41293bba9416cbf56a974c13d0db5, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=418, earliestPutTs=1681606550190 2023-04-16 00:55:51,789 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting a807a8475d49448ea8dba33fc8ab598b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=431, earliestPutTs=1681606550222 2023-04-16 00:55:51,790 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting 1f2310154add4781b73f380f5a9a8949, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=443, earliestPutTs=1681606550858 2023-04-16 00:55:51,796 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp/A/d4bad50742b240f58175e98280b393dd as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/A/d4bad50742b240f58175e98280b393dd 2023-04-16 00:55:51,796 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] throttle.PressureAwareThroughputController(145): 0abd14932d4ec4e5fdc6177f2c7cc55c#C#compaction#467 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-16 00:55:51,804 DEBUG [Thread-3183] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:55:51,805 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:55:51,805 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 0abd14932d4ec4e5fdc6177f2c7cc55c/A of 0abd14932d4ec4e5fdc6177f2c7cc55c into d4bad50742b240f58175e98280b393dd(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 00:55:51,805 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0abd14932d4ec4e5fdc6177f2c7cc55c: 2023-04-16 00:55:51,805 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c., storeName=0abd14932d4ec4e5fdc6177f2c7cc55c/A, priority=13, startTime=1681606551729; duration=0sec 2023-04-16 00:55:51,805 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 00:55:51,808 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:55:51,816 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:55:51,816 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:55:51,816 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:55:51,816 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:51,816 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:55:51,816 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:55:51,816 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:51,816 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:51,818 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:51,818 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:51,818 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:51,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:51,818 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:51,818 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:55:51,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:51,818 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:51,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:55:51,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:51,819 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:55:51,819 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:55:51,819 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:55:51,819 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:55:51,819 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:51,819 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:55:51,820 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:55:51,820 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:55:51,820 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. 2023-04-16 00:55:51,820 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. started... 2023-04-16 00:55:51,820 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 0abd14932d4ec4e5fdc6177f2c7cc55c 3/3 column families, dataSize=127.47 KB heapSize=355.69 KB 2023-04-16 00:55:51,820 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0abd14932d4ec4e5fdc6177f2c7cc55c, store=A 2023-04-16 00:55:51,820 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 00:55:51,820 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0abd14932d4ec4e5fdc6177f2c7cc55c, store=B 2023-04-16 00:55:51,820 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 00:55:51,820 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0abd14932d4ec4e5fdc6177f2c7cc55c, store=C 2023-04-16 00:55:51,820 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 00:55:52,060 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(9158): Flush requested on 0abd14932d4ec4e5fdc6177f2c7cc55c 2023-04-16 00:55:52,060 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. as already flushing 2023-04-16 00:55:52,104 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:52,104 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 161 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57254 deadline: 1681606612092, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:52,105 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:52,105 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 174 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57212 deadline: 1681606612096, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:52,115 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:52,115 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57202 deadline: 1681606612103, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:52,116 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:52,116 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 166 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57268 deadline: 1681606612104, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:52,217 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp/C/9dc03e419bf642f4ae97b52c9b44ef80 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/C/9dc03e419bf642f4ae97b52c9b44ef80 2023-04-16 00:55:52,221 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 0abd14932d4ec4e5fdc6177f2c7cc55c/C of 0abd14932d4ec4e5fdc6177f2c7cc55c into 9dc03e419bf642f4ae97b52c9b44ef80(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 00:55:52,221 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0abd14932d4ec4e5fdc6177f2c7cc55c: 2023-04-16 00:55:52,221 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c., storeName=0abd14932d4ec4e5fdc6177f2c7cc55c/C, priority=13, startTime=1681606551730; duration=0sec 2023-04-16 00:55:52,221 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 00:55:52,223 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:52,223 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 162 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57202 deadline: 1681606612216, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:52,223 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:52,223 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 168 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57268 deadline: 1681606612217, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:52,235 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=480 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp/A/2a22bf6024084919b07d5dc7eafd9f8c 2023-04-16 00:55:52,244 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=480 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp/B/a94f89cd089245449345e46b42f9f598 2023-04-16 00:55:52,257 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=480 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp/C/28e204f9e47f4c5abfb9bd20a16324da 2023-04-16 00:55:52,260 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp/A/2a22bf6024084919b07d5dc7eafd9f8c as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/A/2a22bf6024084919b07d5dc7eafd9f8c 2023-04-16 00:55:52,263 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/A/2a22bf6024084919b07d5dc7eafd9f8c, entries=150, sequenceid=480, filesize=11.9 K 2023-04-16 00:55:52,263 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp/B/a94f89cd089245449345e46b42f9f598 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/B/a94f89cd089245449345e46b42f9f598 2023-04-16 00:55:52,266 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/B/a94f89cd089245449345e46b42f9f598, entries=150, sequenceid=480, filesize=11.9 K 2023-04-16 00:55:52,267 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp/C/28e204f9e47f4c5abfb9bd20a16324da as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/C/28e204f9e47f4c5abfb9bd20a16324da 2023-04-16 00:55:52,270 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/C/28e204f9e47f4c5abfb9bd20a16324da, entries=150, sequenceid=480, filesize=11.9 K 2023-04-16 00:55:52,271 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~355.64 KB/364176, currentSize=53.67 KB/54960 for 0abd14932d4ec4e5fdc6177f2c7cc55c in 451ms, sequenceid=480, compaction requested=false 2023-04-16 00:55:52,271 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 0abd14932d4ec4e5fdc6177f2c7cc55c: 2023-04-16 00:55:52,271 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. 2023-04-16 00:55:52,271 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:55:52,271 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:55:52,271 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:52,271 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:55:52,271 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:55:52,276 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:52,276 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:52,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:52,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:52,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:52,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:52,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:52,278 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:52,278 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:55:52,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:52,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:52,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:52,279 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:55:52,279 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@70eed35c[Count = 0] remaining members to acquire global barrier 2023-04-16 00:55:52,279 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:55:52,279 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:52,281 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:52,281 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:52,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:52,282 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:55:52,282 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:55:52,282 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:55:52,282 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:52,282 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:55:52,283 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:55:52,283 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:52,283 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:52,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:52,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:52,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:52,283 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:55:52,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:52,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:52,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:52,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:52,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:52,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:52,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:52,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:55:52,287 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:55:52,287 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:55:52,287 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:55:52,287 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:55:52,287 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:55:52,287 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:55:52,288 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:52,288 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:52,288 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:52,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:52,288 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:52,288 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:52,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:52,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:52,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:52,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:52,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:52,289 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:52,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:52,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:52,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:52,290 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:52,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:52,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:52,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:52,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:52,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:52,294 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:52,294 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:52,294 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:52,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:52,294 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:52,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:52,294 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:52,294 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:52,294 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:52,294 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:52,294 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:52,294 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:55:52,295 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:55:52,294 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:52,295 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:52,295 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:52,295 DEBUG [Thread-3183] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:55:52,295 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:55:52,295 DEBUG [Thread-3183] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:55:52,395 DEBUG [Thread-3183] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:55:52,396 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:55:52,398 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:55:52,400 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:55:52,400 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:55:52,400 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:55:52,400 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:52,400 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:55:52,400 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:55:52,400 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:52,400 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:52,401 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:52,401 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:52,401 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:52,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:52,402 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:52,402 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:55:52,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:52,405 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:52,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:55:52,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:52,406 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:55:52,406 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:55:52,406 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:55:52,406 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:55:52,406 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:52,406 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:55:52,407 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:55:52,407 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:55:52,407 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. 2023-04-16 00:55:52,407 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. started... 2023-04-16 00:55:52,407 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 0abd14932d4ec4e5fdc6177f2c7cc55c 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-16 00:55:52,408 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0abd14932d4ec4e5fdc6177f2c7cc55c, store=A 2023-04-16 00:55:52,408 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:55:52,408 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0abd14932d4ec4e5fdc6177f2c7cc55c, store=B 2023-04-16 00:55:52,408 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:55:52,408 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0abd14932d4ec4e5fdc6177f2c7cc55c, store=C 2023-04-16 00:55:52,408 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:55:52,417 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=492 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp/A/fab05b92632a4535bb18e46a57bc4e65 2023-04-16 00:55:52,426 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=492 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp/B/26216e227f1a44d2ab0d02b14805c8bd 2023-04-16 00:55:52,429 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(9158): Flush requested on 0abd14932d4ec4e5fdc6177f2c7cc55c 2023-04-16 00:55:52,429 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. as already flushing 2023-04-16 00:55:52,443 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=492 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp/C/9f7e78b544d540e697df8bd15a1f1573 2023-04-16 00:55:52,448 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp/A/fab05b92632a4535bb18e46a57bc4e65 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/A/fab05b92632a4535bb18e46a57bc4e65 2023-04-16 00:55:52,460 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/A/fab05b92632a4535bb18e46a57bc4e65, entries=150, sequenceid=492, filesize=11.9 K 2023-04-16 00:55:52,461 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp/B/26216e227f1a44d2ab0d02b14805c8bd as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/B/26216e227f1a44d2ab0d02b14805c8bd 2023-04-16 00:55:52,464 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/B/26216e227f1a44d2ab0d02b14805c8bd, entries=150, sequenceid=492, filesize=11.9 K 2023-04-16 00:55:52,465 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp/C/9f7e78b544d540e697df8bd15a1f1573 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/C/9f7e78b544d540e697df8bd15a1f1573 2023-04-16 00:55:52,471 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/C/9f7e78b544d540e697df8bd15a1f1573, entries=150, sequenceid=492, filesize=11.9 K 2023-04-16 00:55:52,471 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=40.25 KB/41220 for 0abd14932d4ec4e5fdc6177f2c7cc55c in 64ms, sequenceid=492, compaction requested=true 2023-04-16 00:55:52,473 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 0abd14932d4ec4e5fdc6177f2c7cc55c: 2023-04-16 00:55:52,473 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. 2023-04-16 00:55:52,473 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(9158): Flush requested on 0abd14932d4ec4e5fdc6177f2c7cc55c 2023-04-16 00:55:52,473 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:55:52,473 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:55:52,473 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 0abd14932d4ec4e5fdc6177f2c7cc55c 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-16 00:55:52,473 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:52,474 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:55:52,474 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:55:52,474 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0abd14932d4ec4e5fdc6177f2c7cc55c, store=A 2023-04-16 00:55:52,474 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:55:52,474 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0abd14932d4ec4e5fdc6177f2c7cc55c, store=B 2023-04-16 00:55:52,474 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:55:52,474 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0abd14932d4ec4e5fdc6177f2c7cc55c, store=C 2023-04-16 00:55:52,474 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:55:52,475 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:52,475 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:52,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:52,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:52,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:52,475 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:52,475 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:55:52,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:52,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:52,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:52,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:52,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:52,477 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:55:52,477 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7313921d[Count = 0] remaining members to acquire global barrier 2023-04-16 00:55:52,477 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:55:52,477 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:52,479 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:52,479 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:52,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:52,479 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:52,479 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:55:52,479 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:55:52,479 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:55:52,479 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:55:52,483 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=503 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp/A/e3aa8619e99e4cbf9db6cefc4aab466a 2023-04-16 00:55:52,484 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:55:52,484 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:52,484 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:55:52,485 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:52,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:52,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:52,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:52,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:52,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:52,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:52,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:52,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:52,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:52,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:52,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:55:52,490 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:55:52,490 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:55:52,490 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:55:52,490 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:55:52,490 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:55:52,490 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:55:52,495 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:52,495 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:52,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:52,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:52,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:52,495 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:52,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:52,495 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:52,495 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:52,495 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:52,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:52,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:52,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:52,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:52,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:52,496 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:52,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:52,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:52,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:52,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:52,499 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:52,500 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:52,500 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:52,500 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:52,500 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:52,500 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:52,500 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:52,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:52,500 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:55:52,500 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:55:52,500 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:52,500 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:52,500 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:52,500 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:52,500 DEBUG [Thread-3183] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:55:52,500 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:55:52,500 DEBUG [Thread-3183] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:55:52,500 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:52,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:52,507 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=503 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp/B/24d8518a58b24f75bcbf11fa964162f9 2023-04-16 00:55:52,516 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=503 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp/C/e0682746acfa47f49d5fd48aa5cd6e37 2023-04-16 00:55:52,521 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp/A/e3aa8619e99e4cbf9db6cefc4aab466a as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/A/e3aa8619e99e4cbf9db6cefc4aab466a 2023-04-16 00:55:52,525 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/A/e3aa8619e99e4cbf9db6cefc4aab466a, entries=200, sequenceid=503, filesize=14.3 K 2023-04-16 00:55:52,526 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp/B/24d8518a58b24f75bcbf11fa964162f9 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/B/24d8518a58b24f75bcbf11fa964162f9 2023-04-16 00:55:52,530 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/B/24d8518a58b24f75bcbf11fa964162f9, entries=100, sequenceid=503, filesize=9.5 K 2023-04-16 00:55:52,532 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp/C/e0682746acfa47f49d5fd48aa5cd6e37 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/C/e0682746acfa47f49d5fd48aa5cd6e37 2023-04-16 00:55:52,537 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/C/e0682746acfa47f49d5fd48aa5cd6e37, entries=100, sequenceid=503, filesize=9.5 K 2023-04-16 00:55:52,537 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=107.34 KB/109920 for 0abd14932d4ec4e5fdc6177f2c7cc55c in 64ms, sequenceid=503, compaction requested=true 2023-04-16 00:55:52,538 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 0abd14932d4ec4e5fdc6177f2c7cc55c: 2023-04-16 00:55:52,538 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 00:55:52,538 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-16 00:55:52,540 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-16 00:55:52,540 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 00:55:52,540 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 00:55:52,541 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 46982 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-16 00:55:52,541 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 51862 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-16 00:55:52,541 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HStore(1912): 0abd14932d4ec4e5fdc6177f2c7cc55c/B is initiating minor compaction (all files) 2023-04-16 00:55:52,541 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HStore(1912): 0abd14932d4ec4e5fdc6177f2c7cc55c/A is initiating minor compaction (all files) 2023-04-16 00:55:52,541 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 0abd14932d4ec4e5fdc6177f2c7cc55c/B in TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. 2023-04-16 00:55:52,541 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 0abd14932d4ec4e5fdc6177f2c7cc55c/A in TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. 2023-04-16 00:55:52,541 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/B/bf5fcc1a45b44735aff89fb4837aec89, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/B/a94f89cd089245449345e46b42f9f598, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/B/26216e227f1a44d2ab0d02b14805c8bd, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/B/24d8518a58b24f75bcbf11fa964162f9] into tmpdir=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp, totalSize=45.9 K 2023-04-16 00:55:52,541 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/A/d4bad50742b240f58175e98280b393dd, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/A/2a22bf6024084919b07d5dc7eafd9f8c, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/A/fab05b92632a4535bb18e46a57bc4e65, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/A/e3aa8619e99e4cbf9db6cefc4aab466a] into tmpdir=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp, totalSize=50.6 K 2023-04-16 00:55:52,541 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting bf5fcc1a45b44735aff89fb4837aec89, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=443, earliestPutTs=1681606550858 2023-04-16 00:55:52,541 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.Compactor(207): Compacting d4bad50742b240f58175e98280b393dd, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=443, earliestPutTs=1681606550858 2023-04-16 00:55:52,542 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting a94f89cd089245449345e46b42f9f598, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=480, earliestPutTs=1681606550951 2023-04-16 00:55:52,542 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.Compactor(207): Compacting 2a22bf6024084919b07d5dc7eafd9f8c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=480, earliestPutTs=1681606550951 2023-04-16 00:55:52,542 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting 26216e227f1a44d2ab0d02b14805c8bd, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=492, earliestPutTs=1681606552089 2023-04-16 00:55:52,542 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.Compactor(207): Compacting fab05b92632a4535bb18e46a57bc4e65, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=492, earliestPutTs=1681606552089 2023-04-16 00:55:52,542 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting 24d8518a58b24f75bcbf11fa964162f9, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=503, earliestPutTs=1681606552452 2023-04-16 00:55:52,542 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.Compactor(207): Compacting e3aa8619e99e4cbf9db6cefc4aab466a, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=503, earliestPutTs=1681606552438 2023-04-16 00:55:52,543 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(9158): Flush requested on 0abd14932d4ec4e5fdc6177f2c7cc55c 2023-04-16 00:55:52,543 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 0abd14932d4ec4e5fdc6177f2c7cc55c 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-16 00:55:52,545 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681606552515, max timestamp=1681606552543Num uniques 859; 2023-04-16 00:55:52,545 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-16 00:55:52,545 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681606552515, max timestamp=1681606552543Num uniques 859; 2023-04-16 00:55:52,546 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-16 00:55:52,546 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0abd14932d4ec4e5fdc6177f2c7cc55c, store=A 2023-04-16 00:55:52,546 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 00:55:52,546 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0abd14932d4ec4e5fdc6177f2c7cc55c, store=B 2023-04-16 00:55:52,546 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 00:55:52,546 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0abd14932d4ec4e5fdc6177f2c7cc55c, store=C 2023-04-16 00:55:52,546 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 00:55:52,552 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] throttle.PressureAwareThroughputController(145): 0abd14932d4ec4e5fdc6177f2c7cc55c#B#compaction#478 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-16 00:55:52,552 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] throttle.PressureAwareThroughputController(145): 0abd14932d4ec4e5fdc6177f2c7cc55c#A#compaction#479 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-16 00:55:52,572 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp/B/fb1aef86d13f456f80880e90bbd71fdb as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/B/fb1aef86d13f456f80880e90bbd71fdb 2023-04-16 00:55:52,580 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 0abd14932d4ec4e5fdc6177f2c7cc55c/B of 0abd14932d4ec4e5fdc6177f2c7cc55c into fb1aef86d13f456f80880e90bbd71fdb(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 00:55:52,580 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0abd14932d4ec4e5fdc6177f2c7cc55c: 2023-04-16 00:55:52,580 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c., storeName=0abd14932d4ec4e5fdc6177f2c7cc55c/B, priority=12, startTime=1681606552540; duration=0sec 2023-04-16 00:55:52,580 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 00:55:52,580 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-16 00:55:52,581 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 46982 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-16 00:55:52,581 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HStore(1912): 0abd14932d4ec4e5fdc6177f2c7cc55c/C is initiating minor compaction (all files) 2023-04-16 00:55:52,581 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 0abd14932d4ec4e5fdc6177f2c7cc55c/C in TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. 2023-04-16 00:55:52,581 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/C/9dc03e419bf642f4ae97b52c9b44ef80, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/C/28e204f9e47f4c5abfb9bd20a16324da, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/C/9f7e78b544d540e697df8bd15a1f1573, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/C/e0682746acfa47f49d5fd48aa5cd6e37] into tmpdir=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp, totalSize=45.9 K 2023-04-16 00:55:52,582 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting 9dc03e419bf642f4ae97b52c9b44ef80, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=443, earliestPutTs=1681606550858 2023-04-16 00:55:52,582 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting 28e204f9e47f4c5abfb9bd20a16324da, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=480, earliestPutTs=1681606550951 2023-04-16 00:55:52,582 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting 9f7e78b544d540e697df8bd15a1f1573, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=492, earliestPutTs=1681606552089 2023-04-16 00:55:52,582 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting e0682746acfa47f49d5fd48aa5cd6e37, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=503, earliestPutTs=1681606552452 2023-04-16 00:55:52,589 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] throttle.PressureAwareThroughputController(145): 0abd14932d4ec4e5fdc6177f2c7cc55c#C#compaction#480 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-16 00:55:52,598 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp/C/5353308665c24f2d8d35515f770d600e as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/C/5353308665c24f2d8d35515f770d600e 2023-04-16 00:55:52,600 DEBUG [Thread-3183] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:55:52,601 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:55:52,602 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:55:52,603 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 0abd14932d4ec4e5fdc6177f2c7cc55c/C of 0abd14932d4ec4e5fdc6177f2c7cc55c into 5353308665c24f2d8d35515f770d600e(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 00:55:52,603 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0abd14932d4ec4e5fdc6177f2c7cc55c: 2023-04-16 00:55:52,603 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c., storeName=0abd14932d4ec4e5fdc6177f2c7cc55c/C, priority=12, startTime=1681606552540; duration=0sec 2023-04-16 00:55:52,603 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 00:55:52,603 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:55:52,603 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:55:52,603 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:55:52,603 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:52,603 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:55:52,603 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:55:52,604 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:52,604 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:52,605 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:52,605 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:52,605 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:52,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:52,605 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:52,605 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:55:52,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:52,606 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:52,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:55:52,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:52,606 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:55:52,606 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:55:52,606 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:55:52,606 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:55:52,606 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:52,606 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:55:52,606 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:55:52,606 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:55:52,606 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. 2023-04-16 00:55:52,607 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. started... 2023-04-16 00:55:52,607 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. as already flushing 2023-04-16 00:55:52,607 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. 2023-04-16 00:55:52,607 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:55:52,607 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:55:52,607 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:52,607 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:55:52,607 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:55:52,609 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:52,609 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:52,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:52,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:52,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:52,609 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:52,609 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:55:52,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:52,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:52,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:52,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:52,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:52,610 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:55:52,610 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5e1e775b[Count = 0] remaining members to acquire global barrier 2023-04-16 00:55:52,610 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:55:52,610 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:52,611 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:52,612 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:52,612 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:55:52,612 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:52,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:52,612 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:55:52,612 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:55:52,612 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:55:52,613 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:52,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:52,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:52,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:52,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:52,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:52,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:52,614 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:55:52,614 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:52,614 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:55:52,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:52,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:52,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:52,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:52,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:55:52,615 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:55:52,615 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:55:52,615 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:55:52,615 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:55:52,615 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:55:52,615 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:55:52,617 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:52,617 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:52,617 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:52,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:52,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:52,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:52,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:52,617 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:52,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:52,617 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:52,617 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:52,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:52,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:52,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:52,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:52,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:52,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:52,619 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:52,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:52,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:52,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:52,621 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:52,621 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:52,622 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:52,622 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:52,621 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:52,622 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:55:52,622 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:52,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:52,622 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:55:52,622 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:55:52,622 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:52,622 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:52,622 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:52,622 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:52,622 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:52,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:52,622 DEBUG [Thread-3183] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:55:52,622 DEBUG [Thread-3183] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:55:52,622 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:52,627 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:52,627 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 185 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57202 deadline: 1681606612622, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:52,627 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:52,627 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 192 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57268 deadline: 1681606612625, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:52,722 DEBUG [Thread-3183] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:55:52,723 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:55:52,724 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:55:52,725 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:55:52,725 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:55:52,725 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:55:52,725 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:52,725 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:55:52,725 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:55:52,726 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:52,726 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:52,728 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:52,728 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:52,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:52,728 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:52,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:52,729 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:52,729 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:55:52,729 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:52,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:55:52,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:52,729 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:55:52,729 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:55:52,729 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:55:52,730 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:55:52,730 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:52,730 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:55:52,730 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:55:52,730 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:55:52,730 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. 2023-04-16 00:55:52,730 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. started... 2023-04-16 00:55:52,730 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. as already flushing 2023-04-16 00:55:52,730 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. 2023-04-16 00:55:52,730 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:55:52,730 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:55:52,730 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:52,730 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:55:52,730 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:55:52,732 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:52,732 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 187 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57202 deadline: 1681606612728, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:52,734 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:52,734 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 194 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57268 deadline: 1681606612728, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:52,739 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:52,739 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:52,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:52,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:52,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:52,739 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:52,739 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:55:52,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:52,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:52,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:52,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:52,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:52,741 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:55:52,741 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@230a50c0[Count = 0] remaining members to acquire global barrier 2023-04-16 00:55:52,741 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:55:52,741 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:52,743 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:52,743 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:52,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:52,743 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:55:52,743 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:55:52,743 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:52,743 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:55:52,743 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:55:52,744 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:52,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:52,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:52,744 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:55:52,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:52,744 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:52,744 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:55:52,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:52,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:52,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:52,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:52,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:52,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:52,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:52,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:55:52,748 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:55:52,748 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:55:52,748 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:55:52,748 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:55:52,748 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:55:52,748 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:55:52,749 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:52,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:52,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:52,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:52,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:52,749 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:52,750 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:52,750 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:52,750 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:52,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:52,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:52,750 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:52,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:52,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:52,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:52,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:52,751 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:52,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:52,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:52,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:52,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:52,752 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:52,753 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:52,753 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:52,753 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:52,753 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:52,753 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:52,753 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:55:52,753 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:55:52,753 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:52,753 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:52,753 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:52,753 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:52,753 DEBUG [Thread-3183] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:55:52,753 DEBUG [Thread-3183] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:55:52,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:52,753 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:52,753 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:55:52,753 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:52,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:52,853 DEBUG [Thread-3183] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:55:52,854 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:55:52,855 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:55:52,857 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:55:52,857 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:55:52,857 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:55:52,857 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:52,857 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:55:52,857 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:55:52,857 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:52,857 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:52,860 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:52,860 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:52,860 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:52,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:52,860 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:52,860 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:55:52,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:52,861 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:52,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:55:52,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:52,861 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:55:52,861 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:55:52,861 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:55:52,862 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:55:52,862 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:52,862 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:55:52,862 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:55:52,862 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:55:52,862 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. 2023-04-16 00:55:52,862 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. started... 2023-04-16 00:55:52,862 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. as already flushing 2023-04-16 00:55:52,862 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. 2023-04-16 00:55:52,862 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:55:52,862 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:55:52,862 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:52,862 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:55:52,862 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:55:52,863 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:52,863 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:52,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:52,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:52,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:52,864 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:52,864 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:55:52,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:52,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:52,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:52,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:52,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:52,865 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:55:52,865 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@25f803ce[Count = 0] remaining members to acquire global barrier 2023-04-16 00:55:52,865 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:55:52,866 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:52,867 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:52,867 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:52,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:52,867 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:55:52,867 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:55:52,867 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:55:52,867 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:52,867 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:55:52,869 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:55:52,869 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:52,869 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:52,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:52,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:52,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:52,869 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:55:52,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:52,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:52,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:52,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:52,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:52,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:52,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:52,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:55:52,872 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:55:52,872 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:55:52,872 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:55:52,872 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:55:52,872 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:55:52,872 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:55:52,874 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:52,874 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:52,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:52,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:52,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:52,874 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:52,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:52,874 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:52,875 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:52,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:52,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:52,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:52,876 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:52,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:52,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:52,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:52,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:52,878 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:52,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:52,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:52,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:52,880 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:52,880 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:52,880 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:52,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:52,880 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:52,880 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:55:52,880 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:55:52,880 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:52,880 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:52,881 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:52,881 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:52,881 DEBUG [Thread-3183] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:55:52,881 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:52,881 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:52,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:52,881 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:52,881 DEBUG [Thread-3183] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:55:52,881 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:52,937 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:52,937 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 189 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57202 deadline: 1681606612934, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:52,939 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:52,939 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 196 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57268 deadline: 1681606612936, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:52,967 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=524 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp/A/c66ee34508b5418f8940b8c62c52e14f 2023-04-16 00:55:52,973 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp/A/adaed1c21e2047d785ef510f855c706f as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/A/adaed1c21e2047d785ef510f855c706f 2023-04-16 00:55:52,981 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 0abd14932d4ec4e5fdc6177f2c7cc55c/A of 0abd14932d4ec4e5fdc6177f2c7cc55c into adaed1c21e2047d785ef510f855c706f(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 00:55:52,981 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0abd14932d4ec4e5fdc6177f2c7cc55c: 2023-04-16 00:55:52,981 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c., storeName=0abd14932d4ec4e5fdc6177f2c7cc55c/A, priority=12, startTime=1681606552538; duration=0sec 2023-04-16 00:55:52,981 DEBUG [Thread-3183] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:55:52,981 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 00:55:52,982 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:55:52,983 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:55:52,988 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:55:52,988 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:55:52,988 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:55:52,988 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:52,988 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:55:52,988 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:55:52,988 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:52,989 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:52,990 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:52,990 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:52,990 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:55:52,991 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:52,991 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:52,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:52,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:52,991 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:52,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:55:52,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:52,992 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:55:52,992 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:55:52,992 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:55:52,992 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:55:52,992 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:52,992 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:55:52,993 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:55:52,993 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:55:52,993 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. 2023-04-16 00:55:52,993 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. started... 2023-04-16 00:55:52,993 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. as already flushing 2023-04-16 00:55:52,993 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. 2023-04-16 00:55:52,993 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:55:52,993 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:55:52,993 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:52,993 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:55:52,993 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:55:52,994 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:52,994 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:52,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:52,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:52,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:52,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:52,995 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:52,995 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:55:52,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:52,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:52,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:52,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:52,996 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:55:52,996 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4a94fe16[Count = 0] remaining members to acquire global barrier 2023-04-16 00:55:52,996 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:55:52,996 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:53,001 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:53,001 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:53,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:53,001 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:55:53,001 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:55:53,001 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:55:53,001 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:53,001 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:55:53,002 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:53,002 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:55:53,003 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:53,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:53,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:53,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:53,003 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:55:53,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:53,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:53,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:53,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:53,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:53,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:53,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:53,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:55:53,005 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:55:53,005 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:55:53,005 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:55:53,005 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:55:53,005 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:55:53,005 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:55:53,009 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:53,009 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:53,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:53,009 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:53,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:53,009 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:53,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:53,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:53,009 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:53,009 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:53,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:53,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:53,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:53,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:53,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:53,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:53,010 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:53,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:53,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:53,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:53,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:53,012 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:53,012 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:53,012 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:53,012 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:55:53,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:53,012 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:53,013 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:55:53,012 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:53,012 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:53,013 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:55:53,013 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:53,013 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:53,013 DEBUG [Thread-3183] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:55:53,013 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:53,013 DEBUG [Thread-3183] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:55:53,013 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:53,013 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:53,013 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:53,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:53,113 DEBUG [Thread-3183] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:55:53,113 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:53,113 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 176 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57212 deadline: 1681606613111, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:53,114 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:55:53,115 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:55:53,116 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:55:53,116 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:55:53,116 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:55:53,117 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:53,117 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:55:53,117 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:55:53,117 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:53,117 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:53,117 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:53,117 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57254 deadline: 1681606613114, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:53,120 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:53,120 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:53,120 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:53,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:53,120 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:53,120 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:55:53,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:53,121 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:53,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:55:53,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:53,121 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:55:53,121 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:55:53,122 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:55:53,122 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:55:53,122 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:53,122 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:55:53,122 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:55:53,122 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:55:53,122 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. 2023-04-16 00:55:53,122 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. started... 2023-04-16 00:55:53,122 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. as already flushing 2023-04-16 00:55:53,122 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. 2023-04-16 00:55:53,122 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:55:53,122 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:55:53,122 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:53,122 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:55:53,122 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:55:53,124 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:53,124 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:53,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:53,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:53,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:53,124 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:53,124 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:55:53,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:53,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:53,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:53,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:53,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:53,126 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:55:53,126 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@671decb0[Count = 0] remaining members to acquire global barrier 2023-04-16 00:55:53,126 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:55:53,126 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:53,129 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:53,129 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:53,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:53,129 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:55:53,129 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:55:53,129 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:55:53,129 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:53,129 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:55:53,130 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:53,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:53,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:53,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:53,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:53,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:53,131 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:55:53,131 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:53,131 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:55:53,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:53,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:53,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:53,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:53,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:53,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:55:53,133 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:55:53,133 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:55:53,133 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:55:53,133 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:55:53,133 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:55:53,133 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:55:53,135 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:53,135 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:53,135 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:53,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:53,135 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:53,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:53,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:53,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:53,135 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:53,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:53,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:53,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:53,135 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:53,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:53,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:53,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:53,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:53,137 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:53,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:53,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:53,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:53,143 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:53,143 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:53,143 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:53,143 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:53,143 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:53,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:53,143 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:53,143 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:55:53,143 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:55:53,143 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:53,143 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:53,143 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:53,143 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:53,143 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:53,143 DEBUG [Thread-3183] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:55:53,143 DEBUG [Thread-3183] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:55:53,143 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:53,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:53,242 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:53,242 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 191 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57202 deadline: 1681606613238, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:53,244 DEBUG [Thread-3183] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:55:53,244 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:53,244 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 198 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57268 deadline: 1681606613241, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:53,244 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:55:53,246 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:55:53,247 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:55:53,247 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:55:53,247 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:55:53,247 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:53,247 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:55:53,247 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:55:53,247 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:53,248 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:53,249 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:53,249 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:53,249 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:55:53,250 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:53,250 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:53,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:53,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:53,251 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:53,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:55:53,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:53,251 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:55:53,251 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:55:53,251 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:55:53,251 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:55:53,251 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:53,251 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:55:53,251 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:55:53,252 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:55:53,252 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. 2023-04-16 00:55:53,252 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. started... 2023-04-16 00:55:53,252 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. as already flushing 2023-04-16 00:55:53,252 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. 2023-04-16 00:55:53,252 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:55:53,252 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:55:53,252 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:53,252 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:55:53,252 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:55:53,253 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:53,253 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:53,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:53,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:53,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:53,254 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:53,254 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:55:53,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:53,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:53,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:53,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:53,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:53,255 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:55:53,255 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@16602cc3[Count = 0] remaining members to acquire global barrier 2023-04-16 00:55:53,255 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:55:53,255 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:53,257 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:53,257 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:53,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:53,257 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:55:53,257 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:55:53,257 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:55:53,257 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:53,257 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:55:53,259 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:53,259 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:55:53,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:53,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:53,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:53,259 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:53,259 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:55:53,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:53,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:53,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:53,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:53,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:53,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:53,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:53,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:55:53,263 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:55:53,263 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:55:53,263 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:55:53,263 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:55:53,263 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:55:53,263 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:55:53,264 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:53,264 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:53,265 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:53,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:53,264 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:53,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:53,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:53,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:53,265 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:53,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:53,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:53,265 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:53,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:53,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:53,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:53,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:53,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:53,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:53,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:53,268 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:53,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:53,271 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:53,271 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:53,271 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:53,271 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:53,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:53,271 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:53,271 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:53,271 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:55:53,272 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:55:53,271 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:53,272 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:53,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:53,272 DEBUG [Thread-3183] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:55:53,272 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:53,272 DEBUG [Thread-3183] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:55:53,272 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:53,272 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:53,272 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:53,372 DEBUG [Thread-3183] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:55:53,373 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:55:53,374 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:55:53,375 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:55:53,375 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:55:53,375 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:55:53,375 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:53,375 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:55:53,375 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:55:53,376 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:53,376 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:53,377 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:53,377 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:53,377 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:53,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:53,377 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:53,377 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:55:53,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:53,378 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:53,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:55:53,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:53,378 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:55:53,378 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:55:53,378 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:55:53,379 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:55:53,379 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:53,379 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:55:53,379 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:55:53,379 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:55:53,379 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. 2023-04-16 00:55:53,379 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. started... 2023-04-16 00:55:53,379 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. as already flushing 2023-04-16 00:55:53,379 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. 2023-04-16 00:55:53,379 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:55:53,379 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:55:53,379 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:53,379 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:55:53,379 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:55:53,382 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:53,382 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:53,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:53,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:53,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:53,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:53,384 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:53,384 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:55:53,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:53,384 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=524 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp/B/9dfaeedb80404bbe934f59ebf0eb6615 2023-04-16 00:55:53,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:53,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:53,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:53,385 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:55:53,385 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7363bbb2[Count = 0] remaining members to acquire global barrier 2023-04-16 00:55:53,385 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:55:53,385 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:53,387 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:53,387 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:53,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:53,387 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:53,387 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:55:53,387 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:55:53,387 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:55:53,387 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:55:53,388 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:55:53,388 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:53,388 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:55:53,388 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:53,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:53,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:53,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:53,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:53,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:53,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:53,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:53,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:53,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:53,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:53,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:55:53,393 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:55:53,393 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:55:53,394 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:55:53,394 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:55:53,394 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:55:53,394 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:55:53,395 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:53,395 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:53,395 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:53,395 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:53,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:53,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:53,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:53,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:53,396 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:53,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:53,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:53,396 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:53,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:53,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:53,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:53,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:53,397 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:53,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:53,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:53,398 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=524 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp/C/b686eca7a3dc42439d8000f56b8592bb 2023-04-16 00:55:53,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:53,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:53,402 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp/A/c66ee34508b5418f8940b8c62c52e14f as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/A/c66ee34508b5418f8940b8c62c52e14f 2023-04-16 00:55:53,404 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:53,404 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:53,405 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:53,405 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:53,405 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:53,405 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:53,405 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:53,405 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:53,405 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:53,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:53,405 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:53,405 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:55:53,405 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:55:53,405 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:53,405 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/A/c66ee34508b5418f8940b8c62c52e14f, entries=200, sequenceid=524, filesize=14.3 K 2023-04-16 00:55:53,405 DEBUG [Thread-3183] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:55:53,405 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:55:53,405 DEBUG [Thread-3183] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:55:53,406 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:53,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:53,406 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp/B/9dfaeedb80404bbe934f59ebf0eb6615 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/B/9dfaeedb80404bbe934f59ebf0eb6615 2023-04-16 00:55:53,410 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/B/9dfaeedb80404bbe934f59ebf0eb6615, entries=150, sequenceid=524, filesize=11.9 K 2023-04-16 00:55:53,411 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp/C/b686eca7a3dc42439d8000f56b8592bb as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/C/b686eca7a3dc42439d8000f56b8592bb 2023-04-16 00:55:53,414 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/C/b686eca7a3dc42439d8000f56b8592bb, entries=150, sequenceid=524, filesize=11.9 K 2023-04-16 00:55:53,415 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~184.44 KB/188864, currentSize=114.05 KB/116790 for 0abd14932d4ec4e5fdc6177f2c7cc55c in 872ms, sequenceid=524, compaction requested=false 2023-04-16 00:55:53,415 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 0abd14932d4ec4e5fdc6177f2c7cc55c: 2023-04-16 00:55:53,506 DEBUG [Thread-3183] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:55:53,506 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:55:53,508 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:55:53,509 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:55:53,509 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:55:53,509 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:55:53,509 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:53,509 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:55:53,509 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:55:53,510 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:53,510 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:53,511 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:53,511 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:53,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:53,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:53,511 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:53,511 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:53,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:55:53,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:53,512 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:53,512 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:55:53,512 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:55:53,512 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:55:53,512 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:55:53,512 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:55:53,512 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:53,513 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:55:53,513 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:55:53,513 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:55:53,513 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. 2023-04-16 00:55:53,513 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. started... 2023-04-16 00:55:53,513 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 0abd14932d4ec4e5fdc6177f2c7cc55c 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-16 00:55:53,513 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0abd14932d4ec4e5fdc6177f2c7cc55c, store=A 2023-04-16 00:55:53,513 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:55:53,513 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0abd14932d4ec4e5fdc6177f2c7cc55c, store=B 2023-04-16 00:55:53,513 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:55:53,513 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0abd14932d4ec4e5fdc6177f2c7cc55c, store=C 2023-04-16 00:55:53,513 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:55:53,522 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=547 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp/A/cfc31dc4c0f04e1b8bb19ad5917e9b08 2023-04-16 00:55:53,534 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=547 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp/B/1777f7490a1d429e9e462cc26a768f9f 2023-04-16 00:55:53,544 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=547 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp/C/17eef9ca1bc04f7f8e07b165c7772b52 2023-04-16 00:55:53,548 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp/A/cfc31dc4c0f04e1b8bb19ad5917e9b08 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/A/cfc31dc4c0f04e1b8bb19ad5917e9b08 2023-04-16 00:55:53,552 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/A/cfc31dc4c0f04e1b8bb19ad5917e9b08, entries=150, sequenceid=547, filesize=11.9 K 2023-04-16 00:55:53,553 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp/B/1777f7490a1d429e9e462cc26a768f9f as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/B/1777f7490a1d429e9e462cc26a768f9f 2023-04-16 00:55:53,557 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/B/1777f7490a1d429e9e462cc26a768f9f, entries=150, sequenceid=547, filesize=11.9 K 2023-04-16 00:55:53,558 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp/C/17eef9ca1bc04f7f8e07b165c7772b52 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/C/17eef9ca1bc04f7f8e07b165c7772b52 2023-04-16 00:55:53,561 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/C/17eef9ca1bc04f7f8e07b165c7772b52, entries=150, sequenceid=547, filesize=11.9 K 2023-04-16 00:55:53,562 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=0 B/0 for 0abd14932d4ec4e5fdc6177f2c7cc55c in 49ms, sequenceid=547, compaction requested=true 2023-04-16 00:55:53,562 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 0abd14932d4ec4e5fdc6177f2c7cc55c: 2023-04-16 00:55:53,562 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. 2023-04-16 00:55:53,562 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:55:53,562 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:55:53,562 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:53,562 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:55:53,562 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:55:53,563 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:53,563 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:53,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:53,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:53,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:53,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:53,564 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:53,564 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:55:53,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:53,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:53,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:53,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:53,565 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:55:53,565 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1e6a8f86[Count = 0] remaining members to acquire global barrier 2023-04-16 00:55:53,565 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:55:53,565 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:53,567 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:53,567 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:53,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:53,567 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:53,568 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:55:53,568 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:55:53,568 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:55:53,568 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:55:53,574 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:55:53,574 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:53,574 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:53,574 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:55:53,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:53,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:53,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:53,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:53,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:53,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:53,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:53,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:53,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:53,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:53,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:55:53,577 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:55:53,577 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:55:53,577 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:55:53,577 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:55:53,577 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:55:53,577 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:55:53,578 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:53,578 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:53,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:53,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:53,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:53,578 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:53,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:53,578 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:53,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:53,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:53,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:53,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:53,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:53,580 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:53,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:53,580 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:53,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:53,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:53,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:53,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:53,581 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:53,583 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:53,583 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:53,583 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:53,583 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:53,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:53,583 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:53,583 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:53,583 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:55:53,583 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:55:53,583 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:53,583 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:53,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:53,583 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:53,583 DEBUG [Thread-3183] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:55:53,583 DEBUG [Thread-3183] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:55:53,583 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:53,583 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:53,584 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:53,684 DEBUG [Thread-3183] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:55:53,684 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:55:53,686 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:55:53,687 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:55:53,687 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:55:53,687 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:55:53,687 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:53,687 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:55:53,687 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:55:53,687 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:53,687 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:53,690 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:53,690 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:53,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:53,690 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:53,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:53,690 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:53,690 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:55:53,690 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:53,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:55:53,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:53,691 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:55:53,691 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:55:53,691 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:55:53,691 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:55:53,691 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:53,691 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:55:53,691 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:55:53,691 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:55:53,691 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. 2023-04-16 00:55:53,692 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. started... 2023-04-16 00:55:53,692 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 0abd14932d4ec4e5fdc6177f2c7cc55c: 2023-04-16 00:55:53,692 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. 2023-04-16 00:55:53,692 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:55:53,692 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:55:53,692 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:53,692 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:55:53,692 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:55:53,693 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:53,693 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:53,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:53,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:53,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:53,694 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:53,694 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:55:53,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:53,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:53,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:53,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:53,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:53,695 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:55:53,695 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@161a8897[Count = 0] remaining members to acquire global barrier 2023-04-16 00:55:53,695 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:55:53,695 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:53,696 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:53,696 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:53,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:53,697 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:53,697 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:55:53,697 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:55:53,697 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:55:53,697 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:55:53,698 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:53,698 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:55:53,698 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:53,698 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:55:53,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:53,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:53,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:53,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:53,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:53,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:53,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:53,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:53,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:53,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:53,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:55:53,700 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:55:53,700 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:55:53,700 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:55:53,700 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:55:53,700 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:55:53,700 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:55:53,704 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:53,704 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:53,704 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:53,704 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:53,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:53,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:53,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:53,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:53,704 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:53,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:53,704 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:53,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:53,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:53,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:53,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:53,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:53,706 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:53,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:53,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:53,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:53,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:53,709 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:53,709 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:53,709 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:53,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:53,709 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:53,709 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:53,709 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:55:53,709 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:55:53,709 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:53,710 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:53,710 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:53,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:53,710 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:53,710 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:53,710 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:53,710 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:53,711 DEBUG [Thread-3183] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:55:53,711 DEBUG [Thread-3183] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:55:53,711 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:55:53,760 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(9158): Flush requested on 0abd14932d4ec4e5fdc6177f2c7cc55c 2023-04-16 00:55:53,760 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 0abd14932d4ec4e5fdc6177f2c7cc55c 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-16 00:55:53,760 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0abd14932d4ec4e5fdc6177f2c7cc55c, store=A 2023-04-16 00:55:53,760 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:55:53,760 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0abd14932d4ec4e5fdc6177f2c7cc55c, store=B 2023-04-16 00:55:53,761 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:55:53,761 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0abd14932d4ec4e5fdc6177f2c7cc55c, store=C 2023-04-16 00:55:53,761 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:55:53,768 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=558 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp/A/1b751a155a694b6abf2cb0d6440c07df 2023-04-16 00:55:53,811 DEBUG [Thread-3183] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:55:53,811 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:55:53,813 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:55:53,813 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:55:53,814 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:55:53,814 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:55:53,814 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:53,814 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:55:53,814 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:55:53,814 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:53,814 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:53,815 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:53,815 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:53,815 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:53,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:53,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:53,815 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:53,816 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:55:53,816 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:53,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:55:53,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:53,816 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:55:53,816 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:55:53,816 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:55:53,816 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:55:53,817 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:53,817 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:55:53,817 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:55:53,817 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:55:53,817 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. 2023-04-16 00:55:53,817 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. started... 2023-04-16 00:55:53,817 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. as already flushing 2023-04-16 00:55:53,817 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. 2023-04-16 00:55:53,817 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:55:53,817 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:55:53,817 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:53,817 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:55:53,817 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:55:53,818 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:53,818 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:53,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:53,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:53,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:53,819 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:53,819 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:55:53,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:53,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:53,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:53,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:53,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:53,820 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:55:53,820 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@10ba8a3[Count = 0] remaining members to acquire global barrier 2023-04-16 00:55:53,820 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:55:53,820 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:53,823 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681606553807, max timestamp=1681606553821Num uniques 859; 2023-04-16 00:55:53,823 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-16 00:55:53,824 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681606553807, max timestamp=1681606553821Num uniques 859; 2023-04-16 00:55:53,824 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681606553807, max timestamp=1681606553821Num uniques 859; 2023-04-16 00:55:53,824 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-16 00:55:53,824 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-16 00:55:53,824 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:53,824 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:53,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:53,824 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:55:53,824 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:55:53,824 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:55:53,824 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:53,824 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:55:53,825 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:53,825 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:55:53,825 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:53,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:53,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:53,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:53,825 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:55:53,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:53,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:53,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:53,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:53,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:53,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:53,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:53,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:55:53,827 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:55:53,827 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:55:53,828 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:55:53,828 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:55:53,828 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:55:53,828 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:55:53,829 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:53,829 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:53,829 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:53,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:53,829 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:53,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:53,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:53,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:53,829 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:53,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:53,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:53,830 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:53,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:53,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:53,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:53,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:53,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:53,831 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:53,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:53,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:53,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:53,833 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:53,833 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:53,833 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:53,833 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:53,833 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:55:53,833 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:53,833 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:55:53,833 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:53,833 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:53,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:53,833 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:53,833 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:53,833 DEBUG [Thread-3183] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:55:53,834 DEBUG [Thread-3183] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:55:53,833 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:53,834 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:53,834 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:53,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:53,917 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:53,917 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 213 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57202 deadline: 1681606613910, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:53,918 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:53,918 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 220 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57268 deadline: 1681606613914, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:53,934 DEBUG [Thread-3183] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:55:53,934 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:55:53,936 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:55:53,937 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:55:53,937 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:55:53,937 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:55:53,937 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:53,937 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:55:53,937 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:55:53,938 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:53,938 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:53,942 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:53,942 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:53,942 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:53,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:53,942 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:53,942 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:55:53,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:53,943 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:53,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:55:53,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:53,943 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:55:53,943 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:55:53,943 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:55:53,943 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:55:53,944 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:53,944 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:55:53,944 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:55:53,944 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:55:53,944 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. 2023-04-16 00:55:53,944 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. started... 2023-04-16 00:55:53,944 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. as already flushing 2023-04-16 00:55:53,944 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. 2023-04-16 00:55:53,944 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:55:53,944 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:55:53,944 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:53,944 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:55:53,944 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:55:53,946 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:53,946 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:53,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:53,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:53,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:53,946 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:53,946 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:55:53,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:53,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:53,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:53,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:53,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:53,947 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:55:53,947 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6f5b2d1b[Count = 0] remaining members to acquire global barrier 2023-04-16 00:55:53,947 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:55:53,948 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:53,949 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:53,949 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:53,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:53,949 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:55:53,949 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:55:53,949 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:55:53,949 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:53,949 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:55:53,950 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:55:53,950 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:53,950 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:53,950 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:55:53,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:53,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:53,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:53,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:53,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:53,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:53,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:53,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:53,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:53,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:53,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:55:53,953 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:55:53,953 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:55:53,953 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:55:53,953 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:55:53,953 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:55:53,953 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:55:53,957 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:53,957 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:53,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:53,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:53,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:53,957 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:53,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:53,957 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:53,957 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:53,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:53,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:53,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:53,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:53,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:53,958 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:53,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:53,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:53,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:53,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:53,960 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:53,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:53,965 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:53,965 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:53,965 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:53,965 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:53,965 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:53,965 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:55:53,965 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:55:53,965 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:53,965 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:53,966 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:53,966 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:53,965 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:53,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:53,965 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:53,966 DEBUG [Thread-3183] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:55:53,966 DEBUG [Thread-3183] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:55:53,966 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:53,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:54,023 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:54,023 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 215 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57202 deadline: 1681606614018, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:54,024 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:54,024 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 222 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57268 deadline: 1681606614019, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:54,066 DEBUG [Thread-3183] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:55:54,077 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:55:54,079 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:55:54,080 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:55:54,080 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:55:54,080 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:55:54,080 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:54,080 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:55:54,080 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:55:54,080 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:54,080 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:54,081 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:54,081 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:54,081 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:54,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:54,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:54,082 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:54,082 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:55:54,082 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:54,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:55:54,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:54,082 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:55:54,083 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:55:54,083 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:55:54,083 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:54,083 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:55:54,083 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:55:54,083 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:55:54,083 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. 2023-04-16 00:55:54,083 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. started... 2023-04-16 00:55:54,083 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. as already flushing 2023-04-16 00:55:54,083 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. 2023-04-16 00:55:54,083 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:55:54,083 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:55:54,083 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:54,083 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:55:54,083 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:55:54,085 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:54,085 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:54,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:54,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:54,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:54,085 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:54,085 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:55:54,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:54,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:54,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:54,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:54,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:54,086 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:55:54,086 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@258789c9[Count = 0] remaining members to acquire global barrier 2023-04-16 00:55:54,086 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:55:54,086 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:54,088 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:54,088 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:54,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:54,089 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:55:54,089 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:55:54,089 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:55:54,089 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:54,089 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:55:54,090 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:54,090 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:55:54,090 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:54,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:54,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:54,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:54,090 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:55:54,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:54,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:54,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:54,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:54,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:54,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:54,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:54,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:55:54,092 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:55:54,092 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:55:54,092 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:55:54,092 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:55:54,092 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:55:54,092 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:55:54,093 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:54,093 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:54,094 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:54,094 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:54,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:54,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:54,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:54,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:54,094 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:54,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:54,094 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:54,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:54,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:54,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:54,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:54,095 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:54,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:54,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:54,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:54,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:54,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:54,098 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:54,098 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:54,098 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:54,098 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:54,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:54,098 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:55:54,099 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:55:54,098 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:54,099 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:54,099 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:54,099 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:54,099 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:54,099 DEBUG [Thread-3183] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:55:54,099 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:55:54,099 DEBUG [Thread-3183] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:55:54,099 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:54,099 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:54,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:54,099 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:54,178 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=558 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp/B/72e46b98fcd348ee9b95584e587f3579 2023-04-16 00:55:54,188 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=558 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp/C/38f2c1f13fc14be08a53bc2a04e8cb45 2023-04-16 00:55:54,191 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp/A/1b751a155a694b6abf2cb0d6440c07df as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/A/1b751a155a694b6abf2cb0d6440c07df 2023-04-16 00:55:54,195 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/A/1b751a155a694b6abf2cb0d6440c07df, entries=300, sequenceid=558, filesize=19.0 K 2023-04-16 00:55:54,196 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp/B/72e46b98fcd348ee9b95584e587f3579 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/B/72e46b98fcd348ee9b95584e587f3579 2023-04-16 00:55:54,199 DEBUG [Thread-3183] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:55:54,199 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/B/72e46b98fcd348ee9b95584e587f3579, entries=150, sequenceid=558, filesize=11.9 K 2023-04-16 00:55:54,200 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:55:54,200 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp/C/38f2c1f13fc14be08a53bc2a04e8cb45 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/C/38f2c1f13fc14be08a53bc2a04e8cb45 2023-04-16 00:55:54,201 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:55:54,202 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:55:54,202 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:55:54,202 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:55:54,202 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:54,202 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:55:54,202 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:55:54,203 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:54,203 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:54,203 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/C/38f2c1f13fc14be08a53bc2a04e8cb45, entries=150, sequenceid=558, filesize=11.9 K 2023-04-16 00:55:54,204 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 0abd14932d4ec4e5fdc6177f2c7cc55c in 444ms, sequenceid=558, compaction requested=true 2023-04-16 00:55:54,204 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 0abd14932d4ec4e5fdc6177f2c7cc55c: 2023-04-16 00:55:54,204 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 00:55:54,204 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 00:55:54,204 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 00:55:54,204 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-16 00:55:54,204 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-16 00:55:54,205 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:54,205 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:54,205 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:54,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:54,205 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:54,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:54,205 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:55:54,206 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:54,206 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49562 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-16 00:55:54,206 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HStore(1912): 0abd14932d4ec4e5fdc6177f2c7cc55c/B is initiating minor compaction (all files) 2023-04-16 00:55:54,206 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 59322 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-16 00:55:54,206 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 0abd14932d4ec4e5fdc6177f2c7cc55c/B in TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. 2023-04-16 00:55:54,206 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HStore(1912): 0abd14932d4ec4e5fdc6177f2c7cc55c/A is initiating minor compaction (all files) 2023-04-16 00:55:54,206 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/B/fb1aef86d13f456f80880e90bbd71fdb, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/B/9dfaeedb80404bbe934f59ebf0eb6615, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/B/1777f7490a1d429e9e462cc26a768f9f, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/B/72e46b98fcd348ee9b95584e587f3579] into tmpdir=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp, totalSize=48.4 K 2023-04-16 00:55:54,206 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 0abd14932d4ec4e5fdc6177f2c7cc55c/A in TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. 2023-04-16 00:55:54,206 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/A/adaed1c21e2047d785ef510f855c706f, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/A/c66ee34508b5418f8940b8c62c52e14f, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/A/cfc31dc4c0f04e1b8bb19ad5917e9b08, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/A/1b751a155a694b6abf2cb0d6440c07df] into tmpdir=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp, totalSize=57.9 K 2023-04-16 00:55:54,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:55:54,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:54,206 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:55:54,206 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting fb1aef86d13f456f80880e90bbd71fdb, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=503, earliestPutTs=1681606552089 2023-04-16 00:55:54,206 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:55:54,207 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.Compactor(207): Compacting adaed1c21e2047d785ef510f855c706f, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=503, earliestPutTs=1681606552089 2023-04-16 00:55:54,207 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:55:54,207 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:55:54,207 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:54,207 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting 9dfaeedb80404bbe934f59ebf0eb6615, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=524, earliestPutTs=1681606552515 2023-04-16 00:55:54,207 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:55:54,207 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.Compactor(207): Compacting c66ee34508b5418f8940b8c62c52e14f, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=524, earliestPutTs=1681606552515 2023-04-16 00:55:54,207 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:55:54,207 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:55:54,207 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting 1777f7490a1d429e9e462cc26a768f9f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=547, earliestPutTs=1681606552588 2023-04-16 00:55:54,207 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. 2023-04-16 00:55:54,207 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. started... 2023-04-16 00:55:54,207 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.Compactor(207): Compacting cfc31dc4c0f04e1b8bb19ad5917e9b08, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=547, earliestPutTs=1681606552588 2023-04-16 00:55:54,207 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 0abd14932d4ec4e5fdc6177f2c7cc55c 3/3 column families, dataSize=134.18 KB heapSize=380.30 KB 2023-04-16 00:55:54,208 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting 72e46b98fcd348ee9b95584e587f3579, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=558, earliestPutTs=1681606553752 2023-04-16 00:55:54,208 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0abd14932d4ec4e5fdc6177f2c7cc55c, store=A 2023-04-16 00:55:54,208 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 00:55:54,208 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0abd14932d4ec4e5fdc6177f2c7cc55c, store=B 2023-04-16 00:55:54,208 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.Compactor(207): Compacting 1b751a155a694b6abf2cb0d6440c07df, keycount=300, bloomtype=ROW, size=19.0 K, encoding=NONE, compression=NONE, seqNum=558, earliestPutTs=1681606553747 2023-04-16 00:55:54,208 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 00:55:54,208 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0abd14932d4ec4e5fdc6177f2c7cc55c, store=C 2023-04-16 00:55:54,208 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 00:55:54,222 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] throttle.PressureAwareThroughputController(145): 0abd14932d4ec4e5fdc6177f2c7cc55c#A#compaction#490 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-16 00:55:54,226 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(9158): Flush requested on 0abd14932d4ec4e5fdc6177f2c7cc55c 2023-04-16 00:55:54,226 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. as already flushing 2023-04-16 00:55:54,229 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] throttle.PressureAwareThroughputController(145): 0abd14932d4ec4e5fdc6177f2c7cc55c#B#compaction#491 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-16 00:55:54,237 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=594 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp/A/d249d3f6293540518b0382a8c2e1f37e 2023-04-16 00:55:54,256 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:54,256 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 227 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57268 deadline: 1681606614249, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:54,263 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:54,264 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 221 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57202 deadline: 1681606614256, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:54,265 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp/A/d416eff326974c3bacb78ccaae62834d as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/A/d416eff326974c3bacb78ccaae62834d 2023-04-16 00:55:54,270 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 0abd14932d4ec4e5fdc6177f2c7cc55c/A of 0abd14932d4ec4e5fdc6177f2c7cc55c into d416eff326974c3bacb78ccaae62834d(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 00:55:54,270 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0abd14932d4ec4e5fdc6177f2c7cc55c: 2023-04-16 00:55:54,270 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c., storeName=0abd14932d4ec4e5fdc6177f2c7cc55c/A, priority=12, startTime=1681606554204; duration=0sec 2023-04-16 00:55:54,270 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 00:55:54,270 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-16 00:55:54,271 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49562 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-16 00:55:54,271 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HStore(1912): 0abd14932d4ec4e5fdc6177f2c7cc55c/C is initiating minor compaction (all files) 2023-04-16 00:55:54,271 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 0abd14932d4ec4e5fdc6177f2c7cc55c/C in TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. 2023-04-16 00:55:54,271 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/C/5353308665c24f2d8d35515f770d600e, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/C/b686eca7a3dc42439d8000f56b8592bb, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/C/17eef9ca1bc04f7f8e07b165c7772b52, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/C/38f2c1f13fc14be08a53bc2a04e8cb45] into tmpdir=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp, totalSize=48.4 K 2023-04-16 00:55:54,271 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.Compactor(207): Compacting 5353308665c24f2d8d35515f770d600e, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=503, earliestPutTs=1681606552089 2023-04-16 00:55:54,272 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.Compactor(207): Compacting b686eca7a3dc42439d8000f56b8592bb, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=524, earliestPutTs=1681606552515 2023-04-16 00:55:54,272 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.Compactor(207): Compacting 17eef9ca1bc04f7f8e07b165c7772b52, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=547, earliestPutTs=1681606552588 2023-04-16 00:55:54,272 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.Compactor(207): Compacting 38f2c1f13fc14be08a53bc2a04e8cb45, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=558, earliestPutTs=1681606553752 2023-04-16 00:55:54,285 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] throttle.PressureAwareThroughputController(145): 0abd14932d4ec4e5fdc6177f2c7cc55c#C#compaction#493 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-16 00:55:54,362 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:54,362 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 229 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57268 deadline: 1681606614357, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:54,374 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:54,374 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 223 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57202 deadline: 1681606614365, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:54,569 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:54,569 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 231 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57268 deadline: 1681606614564, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:54,579 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:54,579 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 225 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57202 deadline: 1681606614576, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:54,650 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp/B/2a1d52947afa47a18bff012aa82be4dd as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/B/2a1d52947afa47a18bff012aa82be4dd 2023-04-16 00:55:54,654 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 0abd14932d4ec4e5fdc6177f2c7cc55c/B of 0abd14932d4ec4e5fdc6177f2c7cc55c into 2a1d52947afa47a18bff012aa82be4dd(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 00:55:54,654 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0abd14932d4ec4e5fdc6177f2c7cc55c: 2023-04-16 00:55:54,654 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c., storeName=0abd14932d4ec4e5fdc6177f2c7cc55c/B, priority=12, startTime=1681606554204; duration=0sec 2023-04-16 00:55:54,654 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 00:55:54,662 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=594 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp/B/84f4b2a4eb77486882475d66b97e2de0 2023-04-16 00:55:54,676 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=594 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp/C/1b7876c4bf46450e859584468310e960 2023-04-16 00:55:54,680 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp/A/d249d3f6293540518b0382a8c2e1f37e as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/A/d249d3f6293540518b0382a8c2e1f37e 2023-04-16 00:55:54,684 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/A/d249d3f6293540518b0382a8c2e1f37e, entries=150, sequenceid=594, filesize=11.9 K 2023-04-16 00:55:54,686 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp/B/84f4b2a4eb77486882475d66b97e2de0 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/B/84f4b2a4eb77486882475d66b97e2de0 2023-04-16 00:55:54,689 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/B/84f4b2a4eb77486882475d66b97e2de0, entries=150, sequenceid=594, filesize=11.9 K 2023-04-16 00:55:54,690 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp/C/1b7876c4bf46450e859584468310e960 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/C/1b7876c4bf46450e859584468310e960 2023-04-16 00:55:54,703 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/C/1b7876c4bf46450e859584468310e960, entries=150, sequenceid=594, filesize=11.9 K 2023-04-16 00:55:54,704 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~380.25 KB/389376, currentSize=46.96 KB/48090 for 0abd14932d4ec4e5fdc6177f2c7cc55c in 497ms, sequenceid=594, compaction requested=false 2023-04-16 00:55:54,704 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 0abd14932d4ec4e5fdc6177f2c7cc55c: 2023-04-16 00:55:54,705 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. 2023-04-16 00:55:54,705 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:55:54,705 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:55:54,705 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:54,705 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:55:54,705 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:55:54,706 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp/C/fa1461f5b80647259b51cb623690e569 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/C/fa1461f5b80647259b51cb623690e569 2023-04-16 00:55:54,706 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:54,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:54,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:54,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:54,706 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:54,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:54,707 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:54,707 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:55:54,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:54,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:54,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:54,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:54,709 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:55:54,709 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@679bf3de[Count = 0] remaining members to acquire global barrier 2023-04-16 00:55:54,709 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:55:54,709 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:54,710 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:54,710 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:54,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:54,710 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:55:54,710 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:55:54,710 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:55:54,710 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:54,710 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:55:54,711 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 0abd14932d4ec4e5fdc6177f2c7cc55c/C of 0abd14932d4ec4e5fdc6177f2c7cc55c into fa1461f5b80647259b51cb623690e569(size=12.8 K), total size for store is 24.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 00:55:54,711 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0abd14932d4ec4e5fdc6177f2c7cc55c: 2023-04-16 00:55:54,712 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c., storeName=0abd14932d4ec4e5fdc6177f2c7cc55c/C, priority=12, startTime=1681606554204; duration=0sec 2023-04-16 00:55:54,712 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 00:55:54,712 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:55:54,712 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:54,712 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:55:54,712 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:54,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:54,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:54,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:54,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:54,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:54,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:54,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:54,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:54,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:54,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:54,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:55:54,715 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:55:54,715 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:55:54,715 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:55:54,715 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:55:54,715 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:55:54,715 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:55:54,717 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:54,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:54,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:54,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:54,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:54,718 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:54,718 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:54,718 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:54,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:54,718 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:54,718 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:54,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:54,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:54,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:54,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:54,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:54,719 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:54,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:54,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:54,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:54,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:54,722 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:54,722 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:54,722 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:54,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:54,722 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:54,722 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:54,722 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:54,722 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:54,722 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:54,722 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:54,722 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:54,722 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:54,722 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:54,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:54,722 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:55:54,722 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:55:54,723 DEBUG [Thread-3183] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:55:54,723 DEBUG [Thread-3183] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:55:54,823 DEBUG [Thread-3183] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:55:54,823 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:55:54,825 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:55:54,826 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:55:54,826 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:55:54,826 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:55:54,826 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:54,826 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:55:54,826 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:55:54,826 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:54,827 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:54,828 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:54,828 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:54,828 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:54,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:54,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:54,829 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:54,829 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:54,829 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:55:54,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:55:54,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:54,829 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:55:54,829 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:55:54,829 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:55:54,829 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:55:54,829 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:54,829 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:55:54,830 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:55:54,830 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:55:54,830 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. 2023-04-16 00:55:54,830 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. started... 2023-04-16 00:55:54,830 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 0abd14932d4ec4e5fdc6177f2c7cc55c 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-16 00:55:54,830 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0abd14932d4ec4e5fdc6177f2c7cc55c, store=A 2023-04-16 00:55:54,830 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:55:54,830 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0abd14932d4ec4e5fdc6177f2c7cc55c, store=B 2023-04-16 00:55:54,830 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:55:54,830 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0abd14932d4ec4e5fdc6177f2c7cc55c, store=C 2023-04-16 00:55:54,830 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:55:54,838 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=607 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp/A/87af756bff9b4afdbcf96348945ae7c4 2023-04-16 00:55:54,876 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(9158): Flush requested on 0abd14932d4ec4e5fdc6177f2c7cc55c 2023-04-16 00:55:54,876 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. as already flushing 2023-04-16 00:55:54,942 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681606554915, max timestamp=1681606554941Num uniques 859; 2023-04-16 00:55:54,942 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-16 00:55:54,942 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681606554915, max timestamp=1681606554941Num uniques 859; 2023-04-16 00:55:54,943 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-16 00:55:54,943 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681606554915, max timestamp=1681606554941Num uniques 859; 2023-04-16 00:55:54,943 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-16 00:55:55,009 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:55,009 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 243 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57202 deadline: 1681606615002, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:55,013 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:55,013 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 251 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57268 deadline: 1681606615009, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:55,111 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:55,111 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 245 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57202 deadline: 1681606615110, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:55,119 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:55,119 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 253 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57268 deadline: 1681606615114, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:55,126 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:55,126 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 178 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57212 deadline: 1681606615121, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:55,128 DEBUG [Thread-3181] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4166 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c., hostname=jenkins-hbase4.apache.org,34975,1681606419217, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor32.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-16 00:55:55,135 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:55,135 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57254 deadline: 1681606615129, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:55,137 DEBUG [Thread-3175] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4175 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c., hostname=jenkins-hbase4.apache.org,34975,1681606419217, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor32.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-16 00:55:55,249 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=607 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp/B/97d133c09cca4f00b5c7feb64ab1871f 2023-04-16 00:55:55,322 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:55,322 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 247 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57202 deadline: 1681606615313, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:55,325 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:55,325 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 255 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57268 deadline: 1681606615321, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:55,494 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:55,495 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57228 deadline: 1681606615491, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:55,498 DEBUG [Thread-3177] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8183 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c., hostname=jenkins-hbase4.apache.org,34975,1681606419217, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor32.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-16 00:55:55,629 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:55,629 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 249 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57202 deadline: 1681606615624, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:55,631 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:55,631 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 257 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57268 deadline: 1681606615626, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:55,658 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=607 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp/C/c76426a2729047d989b0b20cdd46b39c 2023-04-16 00:55:55,662 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp/A/87af756bff9b4afdbcf96348945ae7c4 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/A/87af756bff9b4afdbcf96348945ae7c4 2023-04-16 00:55:55,666 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/A/87af756bff9b4afdbcf96348945ae7c4, entries=150, sequenceid=607, filesize=11.9 K 2023-04-16 00:55:55,666 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp/B/97d133c09cca4f00b5c7feb64ab1871f as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/B/97d133c09cca4f00b5c7feb64ab1871f 2023-04-16 00:55:55,670 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/B/97d133c09cca4f00b5c7feb64ab1871f, entries=150, sequenceid=607, filesize=11.9 K 2023-04-16 00:55:55,671 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp/C/c76426a2729047d989b0b20cdd46b39c as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/C/c76426a2729047d989b0b20cdd46b39c 2023-04-16 00:55:55,677 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/C/c76426a2729047d989b0b20cdd46b39c, entries=150, sequenceid=607, filesize=11.9 K 2023-04-16 00:55:55,679 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 0abd14932d4ec4e5fdc6177f2c7cc55c in 849ms, sequenceid=607, compaction requested=true 2023-04-16 00:55:55,680 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 0abd14932d4ec4e5fdc6177f2c7cc55c: 2023-04-16 00:55:55,680 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. 2023-04-16 00:55:55,680 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:55:55,680 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:55:55,680 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:55,680 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:55:55,680 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:55:55,681 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:55,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:55,681 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:55,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:55,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:55,682 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:55,682 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:55:55,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:55,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:55,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:55,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:55,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:55,683 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:55:55,683 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3b09113e[Count = 0] remaining members to acquire global barrier 2023-04-16 00:55:55,683 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:55:55,683 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:55,684 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:55,685 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:55,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:55,685 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:55:55,685 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:55:55,685 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:55:55,685 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:55,685 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:55:55,686 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:55:55,686 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:55,686 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:55,686 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:55:55,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:55,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:55,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:55,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:55,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:55,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:55,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:55,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:55,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:55,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:55,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:55:55,689 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:55:55,689 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:55:55,689 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:55:55,689 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:55:55,689 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:55:55,689 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:55:55,691 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:55,691 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:55,691 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:55,691 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:55,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:55,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:55,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:55,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:55,692 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:55,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:55,692 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:55,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:55,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:55,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:55,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:55,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:55,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:55,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:55,695 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:55,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:55,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:55,700 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:55,700 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:55:55,700 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:55,700 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:55,700 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:55,700 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:55,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:55,700 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:55:55,700 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:55,700 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:55,700 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:55,700 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:55,700 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:55,701 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:55,701 DEBUG [Thread-3183] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:55:55,701 DEBUG [Thread-3183] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:55:55,701 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:55,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:55,801 DEBUG [Thread-3183] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:55:55,801 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:55:55,803 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:55:55,804 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:55:55,804 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:55:55,804 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:55:55,804 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:55,804 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:55:55,804 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:55:55,805 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:55,805 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:55,806 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:55,806 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:55,806 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:55,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:55,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:55,807 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:55,807 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:55,807 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:55:55,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:55:55,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:55,807 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:55:55,808 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:55:55,808 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:55:55,808 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:55:55,808 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:55,808 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:55:55,808 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:55:55,808 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:55:55,808 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. 2023-04-16 00:55:55,808 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. started... 2023-04-16 00:55:55,808 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 0abd14932d4ec4e5fdc6177f2c7cc55c 3/3 column families, dataSize=134.18 KB heapSize=384.98 KB 2023-04-16 00:55:55,809 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0abd14932d4ec4e5fdc6177f2c7cc55c, store=A 2023-04-16 00:55:55,809 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 00:55:55,809 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0abd14932d4ec4e5fdc6177f2c7cc55c, store=B 2023-04-16 00:55:55,809 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 00:55:55,809 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0abd14932d4ec4e5fdc6177f2c7cc55c, store=C 2023-04-16 00:55:55,809 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 00:55:56,135 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(9158): Flush requested on 0abd14932d4ec4e5fdc6177f2c7cc55c 2023-04-16 00:55:56,135 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. as already flushing 2023-04-16 00:55:56,163 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:56,163 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 262 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57268 deadline: 1681606616156, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:56,166 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:56,166 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 255 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57202 deadline: 1681606616157, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:56,217 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=644 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp/A/80d19764c04b42689408d9015a17b1e7 2023-04-16 00:55:56,228 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=644 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp/B/977ff4fdd59146deab38df0aabae3663 2023-04-16 00:55:56,268 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:56,268 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 264 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57268 deadline: 1681606616264, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:56,272 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:56,272 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 257 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57202 deadline: 1681606616267, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:56,475 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:56,475 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 266 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57268 deadline: 1681606616470, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:56,479 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:56,479 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 259 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57202 deadline: 1681606616473, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:56,640 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=644 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp/C/c149193146bd47e0910f36ae9e1a0b21 2023-04-16 00:55:56,643 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp/A/80d19764c04b42689408d9015a17b1e7 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/A/80d19764c04b42689408d9015a17b1e7 2023-04-16 00:55:56,646 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/A/80d19764c04b42689408d9015a17b1e7, entries=150, sequenceid=644, filesize=11.9 K 2023-04-16 00:55:56,646 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp/B/977ff4fdd59146deab38df0aabae3663 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/B/977ff4fdd59146deab38df0aabae3663 2023-04-16 00:55:56,649 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/B/977ff4fdd59146deab38df0aabae3663, entries=150, sequenceid=644, filesize=11.9 K 2023-04-16 00:55:56,650 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp/C/c149193146bd47e0910f36ae9e1a0b21 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/C/c149193146bd47e0910f36ae9e1a0b21 2023-04-16 00:55:56,653 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/C/c149193146bd47e0910f36ae9e1a0b21, entries=150, sequenceid=644, filesize=11.9 K 2023-04-16 00:55:56,654 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~384.94 KB/394176, currentSize=46.96 KB/48090 for 0abd14932d4ec4e5fdc6177f2c7cc55c in 845ms, sequenceid=644, compaction requested=true 2023-04-16 00:55:56,654 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 0abd14932d4ec4e5fdc6177f2c7cc55c: 2023-04-16 00:55:56,654 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. 2023-04-16 00:55:56,654 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:55:56,654 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:55:56,654 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:56,654 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:55:56,654 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:55:56,655 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:56,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:56,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:56,655 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:56,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:56,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:56,656 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:56,656 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:55:56,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:56,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:56,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:56,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:56,657 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:55:56,657 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@d8cd0b1[Count = 0] remaining members to acquire global barrier 2023-04-16 00:55:56,657 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:55:56,657 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:56,659 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:56,659 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:56,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:56,659 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:56,660 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:55:56,659 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:55:56,660 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:55:56,660 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:55:56,661 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:56,661 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:55:56,661 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:56,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:56,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:56,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:56,661 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:55:56,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:56,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:56,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:56,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:56,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:56,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:56,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:56,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:55:56,664 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:55:56,664 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:55:56,664 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:55:56,664 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:55:56,664 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:55:56,664 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:55:56,665 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:56,666 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:56,666 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:56,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:56,666 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:56,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:56,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:56,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:56,666 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:56,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:56,666 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:56,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:56,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:56,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:56,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:56,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:56,667 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:56,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:56,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:56,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:56,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:56,670 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:56,670 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:56,670 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:56,670 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:56,670 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:56,670 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:56,670 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:56,670 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:56,670 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:56,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:56,670 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:55:56,670 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:55:56,670 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:55:56,670 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:56,670 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:56,670 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:56,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:56,670 DEBUG [Thread-3183] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:55:56,670 DEBUG [Thread-3183] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:55:56,771 DEBUG [Thread-3183] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:55:56,771 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:55:56,772 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:55:56,774 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:55:56,774 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:55:56,774 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:55:56,774 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:56,774 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:55:56,774 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:55:56,775 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:56,775 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:56,777 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:56,777 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:56,777 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:56,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:56,777 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:56,777 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:55:56,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:56,778 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:56,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:55:56,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:56,778 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:55:56,778 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:55:56,778 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:55:56,778 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:55:56,778 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:56,778 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:55:56,778 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:55:56,778 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:55:56,778 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. 2023-04-16 00:55:56,778 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. started... 2023-04-16 00:55:56,779 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 0abd14932d4ec4e5fdc6177f2c7cc55c 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-16 00:55:56,779 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0abd14932d4ec4e5fdc6177f2c7cc55c, store=A 2023-04-16 00:55:56,779 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:55:56,779 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0abd14932d4ec4e5fdc6177f2c7cc55c, store=B 2023-04-16 00:55:56,779 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:55:56,779 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0abd14932d4ec4e5fdc6177f2c7cc55c, store=C 2023-04-16 00:55:56,779 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:55:56,782 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(9158): Flush requested on 0abd14932d4ec4e5fdc6177f2c7cc55c 2023-04-16 00:55:56,782 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. as already flushing 2023-04-16 00:55:56,854 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681606556817, max timestamp=1681606556853Num uniques 859; 2023-04-16 00:55:56,854 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-16 00:55:56,855 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681606556817, max timestamp=1681606556853Num uniques 859; 2023-04-16 00:55:56,855 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681606556817, max timestamp=1681606556853Num uniques 859; 2023-04-16 00:55:56,855 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-16 00:55:56,855 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-16 00:55:56,923 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:56,924 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 285 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57268 deadline: 1681606616920, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:56,929 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:56,929 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 278 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57202 deadline: 1681606616921, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:57,028 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:57,029 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 287 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57268 deadline: 1681606617025, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:57,036 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:57,037 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 280 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57202 deadline: 1681606617030, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:57,190 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=654 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp/A/7b0c3d5dda444252b5f5ec4b1a0017f9 2023-04-16 00:55:57,201 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=654 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp/B/1ab9f6ad31904a5cb6512bbd61005a1d 2023-04-16 00:55:57,237 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:57,237 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 289 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57268 deadline: 1681606617230, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:57,244 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:57,244 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 282 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57202 deadline: 1681606617239, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:57,545 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:57,546 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 291 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57268 deadline: 1681606617539, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:57,551 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:57,552 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 284 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57202 deadline: 1681606617545, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:57,611 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=654 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp/C/77de08181e874b69a367c28e023526bc 2023-04-16 00:55:57,616 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp/A/7b0c3d5dda444252b5f5ec4b1a0017f9 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/A/7b0c3d5dda444252b5f5ec4b1a0017f9 2023-04-16 00:55:57,619 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/A/7b0c3d5dda444252b5f5ec4b1a0017f9, entries=150, sequenceid=654, filesize=11.9 K 2023-04-16 00:55:57,620 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp/B/1ab9f6ad31904a5cb6512bbd61005a1d as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/B/1ab9f6ad31904a5cb6512bbd61005a1d 2023-04-16 00:55:57,623 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/B/1ab9f6ad31904a5cb6512bbd61005a1d, entries=150, sequenceid=654, filesize=11.9 K 2023-04-16 00:55:57,624 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp/C/77de08181e874b69a367c28e023526bc as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/C/77de08181e874b69a367c28e023526bc 2023-04-16 00:55:57,626 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/C/77de08181e874b69a367c28e023526bc, entries=150, sequenceid=654, filesize=11.9 K 2023-04-16 00:55:57,627 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 0abd14932d4ec4e5fdc6177f2c7cc55c in 848ms, sequenceid=654, compaction requested=true 2023-04-16 00:55:57,627 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 0abd14932d4ec4e5fdc6177f2c7cc55c: 2023-04-16 00:55:57,627 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. 2023-04-16 00:55:57,627 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:55:57,627 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:55:57,627 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:57,627 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:55:57,627 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:55:57,629 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:57,629 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:57,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:57,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:57,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:57,629 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:57,629 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:55:57,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:57,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:57,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:57,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:57,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:57,631 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:55:57,631 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@14dcc908[Count = 0] remaining members to acquire global barrier 2023-04-16 00:55:57,631 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:55:57,631 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:57,632 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:57,632 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:57,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:57,632 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:55:57,632 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:55:57,632 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:55:57,632 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:57,632 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:55:57,633 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:57,633 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:55:57,633 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:57,633 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:55:57,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:57,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:57,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:57,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:57,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:57,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:57,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:57,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:57,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:57,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:57,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:55:57,636 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:55:57,636 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:55:57,636 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:55:57,636 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:55:57,636 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:55:57,636 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:55:57,638 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:57,638 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:57,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:57,638 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:57,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:57,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:57,638 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:57,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:57,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:57,639 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:57,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:57,639 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:57,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:57,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:57,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:57,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:57,640 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:57,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:57,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:57,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:57,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:57,642 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:57,642 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:57,642 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:57,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:57,642 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:57,642 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:57,642 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:57,642 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:57,642 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:57,642 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:57,642 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:57,642 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:57,642 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:55:57,642 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:55:57,642 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:55:57,643 DEBUG [Thread-3183] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:55:57,643 DEBUG [Thread-3183] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:55:57,651 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:57,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:57,743 DEBUG [Thread-3183] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:55:57,743 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:55:57,745 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:55:57,747 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:55:57,747 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:55:57,747 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:55:57,748 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:57,748 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:55:57,748 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:55:57,748 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:57,748 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:57,750 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:57,750 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:57,750 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:57,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:57,750 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:57,750 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:55:57,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:57,750 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:57,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:55:57,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:57,751 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:55:57,751 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:55:57,751 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:55:57,751 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:55:57,751 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:57,751 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:55:57,751 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:55:57,751 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:55:57,752 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. 2023-04-16 00:55:57,752 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. started... 2023-04-16 00:55:57,752 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 0abd14932d4ec4e5fdc6177f2c7cc55c 3/3 column families, dataSize=134.18 KB heapSize=384.98 KB 2023-04-16 00:55:57,752 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0abd14932d4ec4e5fdc6177f2c7cc55c, store=A 2023-04-16 00:55:57,752 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 00:55:57,752 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0abd14932d4ec4e5fdc6177f2c7cc55c, store=B 2023-04-16 00:55:57,752 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 00:55:57,752 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0abd14932d4ec4e5fdc6177f2c7cc55c, store=C 2023-04-16 00:55:57,752 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 00:55:58,057 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(9158): Flush requested on 0abd14932d4ec4e5fdc6177f2c7cc55c 2023-04-16 00:55:58,057 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. as already flushing 2023-04-16 00:55:58,111 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:58,111 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 289 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57202 deadline: 1681606618099, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:58,111 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:58,111 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 297 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57268 deadline: 1681606618099, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:58,170 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=691 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp/A/19242e54d92c46e8a518c507507c154f 2023-04-16 00:55:58,181 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=691 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp/B/96b56dda3b174aaf80e9c8fd418cf46a 2023-04-16 00:55:58,192 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=691 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp/C/0f8c5fb814544febb5bbf68eccd59528 2023-04-16 00:55:58,197 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp/A/19242e54d92c46e8a518c507507c154f as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/A/19242e54d92c46e8a518c507507c154f 2023-04-16 00:55:58,203 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/A/19242e54d92c46e8a518c507507c154f, entries=150, sequenceid=691, filesize=11.9 K 2023-04-16 00:55:58,203 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp/B/96b56dda3b174aaf80e9c8fd418cf46a as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/B/96b56dda3b174aaf80e9c8fd418cf46a 2023-04-16 00:55:58,206 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/B/96b56dda3b174aaf80e9c8fd418cf46a, entries=150, sequenceid=691, filesize=11.9 K 2023-04-16 00:55:58,207 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp/C/0f8c5fb814544febb5bbf68eccd59528 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/C/0f8c5fb814544febb5bbf68eccd59528 2023-04-16 00:55:58,210 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/C/0f8c5fb814544febb5bbf68eccd59528, entries=150, sequenceid=691, filesize=11.9 K 2023-04-16 00:55:58,211 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~384.94 KB/394176, currentSize=46.96 KB/48090 for 0abd14932d4ec4e5fdc6177f2c7cc55c in 459ms, sequenceid=691, compaction requested=true 2023-04-16 00:55:58,211 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 0abd14932d4ec4e5fdc6177f2c7cc55c: 2023-04-16 00:55:58,211 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. 2023-04-16 00:55:58,211 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:55:58,211 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:55:58,211 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:58,211 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:55:58,211 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:55:58,212 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:58,212 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:58,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:58,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:58,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:58,213 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:58,213 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:55:58,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:58,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:58,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:58,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:58,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:58,214 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:55:58,215 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2ce1cd4e[Count = 0] remaining members to acquire global barrier 2023-04-16 00:55:58,215 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:55:58,215 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:58,217 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:58,217 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:58,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:58,217 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:55:58,217 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:55:58,217 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:58,217 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:55:58,217 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:55:58,218 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(9158): Flush requested on 0abd14932d4ec4e5fdc6177f2c7cc55c 2023-04-16 00:55:58,218 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 0abd14932d4ec4e5fdc6177f2c7cc55c 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-16 00:55:58,218 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0abd14932d4ec4e5fdc6177f2c7cc55c, store=A 2023-04-16 00:55:58,218 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:55:58,218 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0abd14932d4ec4e5fdc6177f2c7cc55c, store=B 2023-04-16 00:55:58,218 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:55:58,218 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:58,218 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0abd14932d4ec4e5fdc6177f2c7cc55c, store=C 2023-04-16 00:55:58,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:58,218 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:55:58,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:58,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:58,218 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:55:58,218 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:58,219 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:55:58,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:58,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:58,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:58,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:58,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:58,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:58,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:58,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:55:58,221 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:55:58,221 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:55:58,221 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:55:58,221 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:55:58,221 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:55:58,221 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:55:58,227 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:58,227 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:58,227 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:58,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:58,227 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:58,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:58,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:58,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:58,228 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:58,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:58,228 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:58,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:58,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:58,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:58,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:58,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:58,229 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:58,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:58,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:58,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:58,230 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=702 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp/A/a72707feed22489a88bd032ea60dcfee 2023-04-16 00:55:58,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:58,232 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:58,232 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:58,232 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:58,232 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:55:58,232 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:55:58,232 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:58,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:58,232 DEBUG [Thread-3183] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:55:58,232 DEBUG [Thread-3183] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:55:58,232 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:58,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:58,237 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:58,237 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:58,237 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:58,237 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:58,237 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:58,237 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:58,237 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:58,243 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=702 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp/B/a40971af59fd4d3288b0337f3eea29f4 2023-04-16 00:55:58,251 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=702 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp/C/e20e89b785404b89b92c3d112d8d44c0 2023-04-16 00:55:58,255 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp/A/a72707feed22489a88bd032ea60dcfee as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/A/a72707feed22489a88bd032ea60dcfee 2023-04-16 00:55:58,257 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/A/a72707feed22489a88bd032ea60dcfee, entries=200, sequenceid=702, filesize=14.3 K 2023-04-16 00:55:58,258 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp/B/a40971af59fd4d3288b0337f3eea29f4 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/B/a40971af59fd4d3288b0337f3eea29f4 2023-04-16 00:55:58,263 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/B/a40971af59fd4d3288b0337f3eea29f4, entries=150, sequenceid=702, filesize=11.9 K 2023-04-16 00:55:58,263 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp/C/e20e89b785404b89b92c3d112d8d44c0 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/C/e20e89b785404b89b92c3d112d8d44c0 2023-04-16 00:55:58,268 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/C/e20e89b785404b89b92c3d112d8d44c0, entries=150, sequenceid=702, filesize=11.9 K 2023-04-16 00:55:58,268 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=67.09 KB/68700 for 0abd14932d4ec4e5fdc6177f2c7cc55c in 50ms, sequenceid=702, compaction requested=true 2023-04-16 00:55:58,268 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 0abd14932d4ec4e5fdc6177f2c7cc55c: 2023-04-16 00:55:58,269 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 00:55:58,269 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-04-16 00:55:58,269 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-04-16 00:55:58,269 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 00:55:58,269 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 00:55:58,270 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 7 files of size 86256 starting at candidate #0 after considering 15 permutations with 15 in ratio 2023-04-16 00:55:58,270 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 7 files of size 88696 starting at candidate #0 after considering 15 permutations with 15 in ratio 2023-04-16 00:55:58,270 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HStore(1912): 0abd14932d4ec4e5fdc6177f2c7cc55c/B is initiating minor compaction (all files) 2023-04-16 00:55:58,271 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HStore(1912): 0abd14932d4ec4e5fdc6177f2c7cc55c/A is initiating minor compaction (all files) 2023-04-16 00:55:58,271 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 0abd14932d4ec4e5fdc6177f2c7cc55c/B in TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. 2023-04-16 00:55:58,271 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 0abd14932d4ec4e5fdc6177f2c7cc55c/A in TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. 2023-04-16 00:55:58,271 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/B/2a1d52947afa47a18bff012aa82be4dd, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/B/84f4b2a4eb77486882475d66b97e2de0, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/B/97d133c09cca4f00b5c7feb64ab1871f, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/B/977ff4fdd59146deab38df0aabae3663, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/B/1ab9f6ad31904a5cb6512bbd61005a1d, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/B/96b56dda3b174aaf80e9c8fd418cf46a, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/B/a40971af59fd4d3288b0337f3eea29f4] into tmpdir=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp, totalSize=84.2 K 2023-04-16 00:55:58,271 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/A/d416eff326974c3bacb78ccaae62834d, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/A/d249d3f6293540518b0382a8c2e1f37e, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/A/87af756bff9b4afdbcf96348945ae7c4, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/A/80d19764c04b42689408d9015a17b1e7, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/A/7b0c3d5dda444252b5f5ec4b1a0017f9, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/A/19242e54d92c46e8a518c507507c154f, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/A/a72707feed22489a88bd032ea60dcfee] into tmpdir=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp, totalSize=86.6 K 2023-04-16 00:55:58,271 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting 2a1d52947afa47a18bff012aa82be4dd, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=558, earliestPutTs=1681606553752 2023-04-16 00:55:58,271 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.Compactor(207): Compacting d416eff326974c3bacb78ccaae62834d, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=558, earliestPutTs=1681606553752 2023-04-16 00:55:58,271 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting 84f4b2a4eb77486882475d66b97e2de0, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=594, earliestPutTs=1681606553871 2023-04-16 00:55:58,271 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.Compactor(207): Compacting d249d3f6293540518b0382a8c2e1f37e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=594, earliestPutTs=1681606553871 2023-04-16 00:55:58,272 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting 97d133c09cca4f00b5c7feb64ab1871f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=607, earliestPutTs=1681606554246 2023-04-16 00:55:58,272 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.Compactor(207): Compacting 87af756bff9b4afdbcf96348945ae7c4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=607, earliestPutTs=1681606554246 2023-04-16 00:55:58,272 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting 977ff4fdd59146deab38df0aabae3663, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=644, earliestPutTs=1681606554992 2023-04-16 00:55:58,272 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.Compactor(207): Compacting 80d19764c04b42689408d9015a17b1e7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=644, earliestPutTs=1681606554992 2023-04-16 00:55:58,272 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting 1ab9f6ad31904a5cb6512bbd61005a1d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=654, earliestPutTs=1681606556147 2023-04-16 00:55:58,273 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.Compactor(207): Compacting 7b0c3d5dda444252b5f5ec4b1a0017f9, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=654, earliestPutTs=1681606556147 2023-04-16 00:55:58,273 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting 96b56dda3b174aaf80e9c8fd418cf46a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=691, earliestPutTs=1681606556906 2023-04-16 00:55:58,273 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting a40971af59fd4d3288b0337f3eea29f4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=702, earliestPutTs=1681606558075 2023-04-16 00:55:58,273 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.Compactor(207): Compacting 19242e54d92c46e8a518c507507c154f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=691, earliestPutTs=1681606556906 2023-04-16 00:55:58,274 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.Compactor(207): Compacting a72707feed22489a88bd032ea60dcfee, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=702, earliestPutTs=1681606558066 2023-04-16 00:55:58,274 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(9158): Flush requested on 0abd14932d4ec4e5fdc6177f2c7cc55c 2023-04-16 00:55:58,274 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 0abd14932d4ec4e5fdc6177f2c7cc55c 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-16 00:55:58,275 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0abd14932d4ec4e5fdc6177f2c7cc55c, store=A 2023-04-16 00:55:58,275 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:55:58,275 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0abd14932d4ec4e5fdc6177f2c7cc55c, store=B 2023-04-16 00:55:58,275 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:55:58,275 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0abd14932d4ec4e5fdc6177f2c7cc55c, store=C 2023-04-16 00:55:58,275 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:55:58,290 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] throttle.PressureAwareThroughputController(145): 0abd14932d4ec4e5fdc6177f2c7cc55c#A#compaction#511 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-16 00:55:58,292 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] throttle.PressureAwareThroughputController(145): 0abd14932d4ec4e5fdc6177f2c7cc55c#B#compaction#512 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-16 00:55:58,302 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=716 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp/A/dd7214cf35164c9e97fe1d6b6b2cf90c 2023-04-16 00:55:58,320 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp/A/e493d1145a3e4ffdb15b77bc9b9e00ec as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/A/e493d1145a3e4ffdb15b77bc9b9e00ec 2023-04-16 00:55:58,325 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=716 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp/B/dbb5eea9c8dc47a79a432c684793be13 2023-04-16 00:55:58,325 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in 0abd14932d4ec4e5fdc6177f2c7cc55c/A of 0abd14932d4ec4e5fdc6177f2c7cc55c into e493d1145a3e4ffdb15b77bc9b9e00ec(size=13.1 K), total size for store is 13.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 00:55:58,326 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0abd14932d4ec4e5fdc6177f2c7cc55c: 2023-04-16 00:55:58,326 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c., storeName=0abd14932d4ec4e5fdc6177f2c7cc55c/A, priority=9, startTime=1681606558269; duration=0sec 2023-04-16 00:55:58,326 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 00:55:58,326 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-04-16 00:55:58,326 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp/B/eac3fce8f98b4eaabf995b5dc9ecacb9 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/B/eac3fce8f98b4eaabf995b5dc9ecacb9 2023-04-16 00:55:58,329 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 7 files of size 86256 starting at candidate #0 after considering 15 permutations with 15 in ratio 2023-04-16 00:55:58,329 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HStore(1912): 0abd14932d4ec4e5fdc6177f2c7cc55c/C is initiating minor compaction (all files) 2023-04-16 00:55:58,329 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 0abd14932d4ec4e5fdc6177f2c7cc55c/C in TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. 2023-04-16 00:55:58,330 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/C/fa1461f5b80647259b51cb623690e569, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/C/1b7876c4bf46450e859584468310e960, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/C/c76426a2729047d989b0b20cdd46b39c, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/C/c149193146bd47e0910f36ae9e1a0b21, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/C/77de08181e874b69a367c28e023526bc, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/C/0f8c5fb814544febb5bbf68eccd59528, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/C/e20e89b785404b89b92c3d112d8d44c0] into tmpdir=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp, totalSize=84.2 K 2023-04-16 00:55:58,330 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.Compactor(207): Compacting fa1461f5b80647259b51cb623690e569, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=558, earliestPutTs=1681606553752 2023-04-16 00:55:58,330 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.Compactor(207): Compacting 1b7876c4bf46450e859584468310e960, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=594, earliestPutTs=1681606553871 2023-04-16 00:55:58,332 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.Compactor(207): Compacting c76426a2729047d989b0b20cdd46b39c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=607, earliestPutTs=1681606554246 2023-04-16 00:55:58,332 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in 0abd14932d4ec4e5fdc6177f2c7cc55c/B of 0abd14932d4ec4e5fdc6177f2c7cc55c into eac3fce8f98b4eaabf995b5dc9ecacb9(size=13.1 K), total size for store is 13.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 00:55:58,332 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0abd14932d4ec4e5fdc6177f2c7cc55c: 2023-04-16 00:55:58,332 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c., storeName=0abd14932d4ec4e5fdc6177f2c7cc55c/B, priority=9, startTime=1681606558269; duration=0sec 2023-04-16 00:55:58,332 DEBUG [Thread-3183] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:55:58,332 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 00:55:58,333 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:55:58,333 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.Compactor(207): Compacting c149193146bd47e0910f36ae9e1a0b21, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=644, earliestPutTs=1681606554992 2023-04-16 00:55:58,334 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.Compactor(207): Compacting 77de08181e874b69a367c28e023526bc, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=654, earliestPutTs=1681606556147 2023-04-16 00:55:58,334 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.Compactor(207): Compacting 0f8c5fb814544febb5bbf68eccd59528, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=691, earliestPutTs=1681606556906 2023-04-16 00:55:58,335 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:55:58,335 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.Compactor(207): Compacting e20e89b785404b89b92c3d112d8d44c0, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=702, earliestPutTs=1681606558075 2023-04-16 00:55:58,336 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:55:58,336 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:55:58,336 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:55:58,336 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:58,336 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:55:58,337 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:55:58,337 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:58,337 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:58,340 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=716 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp/C/999b744968634309b19688352170de24 2023-04-16 00:55:58,342 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:58,342 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:58,342 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:58,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:58,342 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:58,342 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:55:58,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:58,342 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:58,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:55:58,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:58,344 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:55:58,344 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:55:58,344 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:55:58,344 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:55:58,344 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:58,344 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:55:58,345 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:55:58,345 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:55:58,345 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. 2023-04-16 00:55:58,345 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. started... 2023-04-16 00:55:58,345 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. as already flushing 2023-04-16 00:55:58,345 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. 2023-04-16 00:55:58,345 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:55:58,345 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:55:58,345 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:58,345 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:55:58,345 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:55:58,346 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:58,346 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:58,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:58,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:58,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:58,347 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:58,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:58,347 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:55:58,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:58,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:58,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:58,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:58,348 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:55:58,348 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5f12a90d[Count = 0] remaining members to acquire global barrier 2023-04-16 00:55:58,348 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:55:58,348 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:58,350 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:58,350 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:58,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:58,350 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:55:58,350 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:55:58,350 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:55:58,350 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:58,350 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp/A/dd7214cf35164c9e97fe1d6b6b2cf90c as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/A/dd7214cf35164c9e97fe1d6b6b2cf90c 2023-04-16 00:55:58,350 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:55:58,353 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:55:58,353 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:58,353 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:58,353 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:55:58,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:58,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:58,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:58,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:58,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:58,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:58,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:58,354 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] throttle.PressureAwareThroughputController(145): 0abd14932d4ec4e5fdc6177f2c7cc55c#C#compaction#515 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-16 00:55:58,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:58,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:58,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:58,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:55:58,356 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:55:58,356 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:55:58,356 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:55:58,356 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:55:58,356 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:55:58,356 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:55:58,357 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/A/dd7214cf35164c9e97fe1d6b6b2cf90c, entries=200, sequenceid=716, filesize=14.3 K 2023-04-16 00:55:58,358 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp/B/dbb5eea9c8dc47a79a432c684793be13 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/B/dbb5eea9c8dc47a79a432c684793be13 2023-04-16 00:55:58,359 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:58,359 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:58,359 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:58,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:58,360 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:58,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:58,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:58,360 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:58,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:58,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:58,360 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:58,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:58,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:58,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:58,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:58,361 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:58,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:58,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:58,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:58,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:58,363 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:58,363 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:58,363 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:58,363 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:58,363 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:58,363 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:58,363 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:58,363 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:55:58,363 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:58,363 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:58,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:58,363 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:58,363 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:58,363 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:55:58,363 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:55:58,364 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:58,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:58,364 DEBUG [Thread-3183] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:55:58,364 DEBUG [Thread-3183] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:55:58,366 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/B/dbb5eea9c8dc47a79a432c684793be13, entries=150, sequenceid=716, filesize=11.9 K 2023-04-16 00:55:58,367 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp/C/999b744968634309b19688352170de24 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/C/999b744968634309b19688352170de24 2023-04-16 00:55:58,369 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:58,369 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 305 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57202 deadline: 1681606618368, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:58,370 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/C/999b744968634309b19688352170de24, entries=150, sequenceid=716, filesize=11.9 K 2023-04-16 00:55:58,371 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for 0abd14932d4ec4e5fdc6177f2c7cc55c in 97ms, sequenceid=716, compaction requested=false 2023-04-16 00:55:58,371 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 0abd14932d4ec4e5fdc6177f2c7cc55c: 2023-04-16 00:55:58,384 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(9158): Flush requested on 0abd14932d4ec4e5fdc6177f2c7cc55c 2023-04-16 00:55:58,384 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 0abd14932d4ec4e5fdc6177f2c7cc55c 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-16 00:55:58,384 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0abd14932d4ec4e5fdc6177f2c7cc55c, store=A 2023-04-16 00:55:58,384 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:55:58,384 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0abd14932d4ec4e5fdc6177f2c7cc55c, store=B 2023-04-16 00:55:58,384 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:55:58,384 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0abd14932d4ec4e5fdc6177f2c7cc55c, store=C 2023-04-16 00:55:58,384 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:55:58,464 DEBUG [Thread-3183] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:55:58,465 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:55:58,466 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:55:58,467 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:55:58,467 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:55:58,467 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:55:58,467 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:58,467 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:55:58,467 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:55:58,467 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:58,467 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:58,469 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:58,469 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:58,469 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:58,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:58,469 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:58,469 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:55:58,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:58,469 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:58,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:55:58,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:58,470 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:55:58,470 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:55:58,470 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:55:58,470 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:55:58,470 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:58,470 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:55:58,470 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:55:58,470 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:55:58,471 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. 2023-04-16 00:55:58,471 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. started... 2023-04-16 00:55:58,471 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. as already flushing 2023-04-16 00:55:58,471 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. 2023-04-16 00:55:58,471 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:55:58,471 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:55:58,471 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:58,471 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:55:58,471 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:55:58,473 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:58,473 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 323 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57268 deadline: 1681606618464, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:58,473 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:58,473 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 307 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57202 deadline: 1681606618471, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:58,475 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:58,475 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:58,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:58,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:58,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:58,475 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:58,475 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:55:58,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:58,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:58,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:58,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:58,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:58,476 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:55:58,476 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@34f0f306[Count = 0] remaining members to acquire global barrier 2023-04-16 00:55:58,476 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:55:58,477 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:58,478 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:58,478 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:58,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:58,478 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:55:58,478 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:55:58,478 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:55:58,478 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:58,478 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:55:58,479 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:55:58,479 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:58,479 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:58,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:58,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:58,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:58,479 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:55:58,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:58,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:58,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:58,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:58,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:58,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:58,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:58,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:55:58,482 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:55:58,482 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:55:58,482 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:55:58,482 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:55:58,482 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:55:58,482 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:55:58,483 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:58,483 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:58,483 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:58,483 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:58,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:58,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:58,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:58,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:58,484 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:58,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:58,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:58,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:58,485 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:58,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:58,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:58,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:58,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:58,486 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:58,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:58,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:58,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:58,493 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:58,493 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:58,493 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:58,493 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:58,493 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:58,493 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:58,493 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:55:58,493 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:58,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:58,493 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:58,493 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:58,493 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:55:58,494 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:55:58,494 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:58,494 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:58,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:58,494 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:58,494 DEBUG [Thread-3183] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:55:58,494 DEBUG [Thread-3183] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:55:58,579 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:58,580 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 325 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57268 deadline: 1681606618574, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:58,594 DEBUG [Thread-3183] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:55:58,595 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:55:58,596 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:55:58,597 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:55:58,598 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:55:58,598 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:55:58,598 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:58,598 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:55:58,598 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:55:58,598 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:58,598 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:58,601 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:58,601 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:58,601 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:58,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:58,601 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:58,602 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:55:58,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:58,602 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:58,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:55:58,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:58,602 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:55:58,603 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:55:58,603 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:55:58,603 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:55:58,603 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:58,603 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:55:58,603 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:55:58,603 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:55:58,603 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. 2023-04-16 00:55:58,603 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. started... 2023-04-16 00:55:58,603 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. as already flushing 2023-04-16 00:55:58,603 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. 2023-04-16 00:55:58,603 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:55:58,603 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:55:58,603 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:58,603 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:55:58,603 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:55:58,605 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:58,605 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:58,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:58,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:58,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:58,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:58,606 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:58,606 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:55:58,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:58,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:58,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:58,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:58,607 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:55:58,607 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4696f819[Count = 0] remaining members to acquire global barrier 2023-04-16 00:55:58,607 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:55:58,607 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:58,608 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:58,608 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:58,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:58,608 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:55:58,608 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:55:58,608 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:55:58,608 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:58,608 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:55:58,610 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:58,610 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:55:58,610 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:58,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:58,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:58,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:58,610 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:55:58,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:58,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:58,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:58,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:58,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:58,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:58,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:58,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:55:58,613 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:55:58,613 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:55:58,613 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:55:58,613 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:55:58,613 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:55:58,613 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:55:58,614 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:58,614 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:58,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:58,614 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:58,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:58,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:58,614 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:58,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:58,615 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:58,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:58,615 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:58,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:58,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:58,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:58,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:58,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:58,616 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:58,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:58,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:58,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:58,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:58,618 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:58,618 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:58,618 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:58,618 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:58,618 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:58,618 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:58,618 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:58,618 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:58,618 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:55:58,618 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:55:58,618 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:55:58,618 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:58,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:58,618 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:58,618 DEBUG [Thread-3183] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:55:58,618 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:58,619 DEBUG [Thread-3183] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:55:58,619 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:58,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:58,678 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:58,678 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 309 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57202 deadline: 1681606618675, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:58,719 DEBUG [Thread-3183] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:55:58,719 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:55:58,721 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:55:58,722 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:55:58,722 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:55:58,722 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:55:58,722 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:58,722 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:55:58,722 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:55:58,724 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:58,724 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:58,725 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:58,725 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:58,725 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:58,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:58,725 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:58,725 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:55:58,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:58,726 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:58,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:55:58,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:58,726 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:55:58,726 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:55:58,727 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:55:58,727 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:55:58,727 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:58,727 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:55:58,727 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:55:58,727 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:55:58,727 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. 2023-04-16 00:55:58,727 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. started... 2023-04-16 00:55:58,727 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. as already flushing 2023-04-16 00:55:58,727 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. 2023-04-16 00:55:58,727 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:55:58,727 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:55:58,727 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:58,727 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:55:58,727 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:55:58,730 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:58,730 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:58,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:58,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:58,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:58,730 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:58,730 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:55:58,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:58,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:58,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:58,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:58,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:58,734 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:55:58,734 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@643fc638[Count = 0] remaining members to acquire global barrier 2023-04-16 00:55:58,734 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:55:58,734 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:58,738 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:58,738 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:58,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:58,738 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:55:58,738 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:55:58,738 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:55:58,738 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:58,738 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:55:58,741 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:58,741 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:55:58,741 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:58,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:58,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:58,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:58,741 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:55:58,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:58,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:58,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:58,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:58,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:58,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:58,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:58,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:55:58,743 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:55:58,743 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:55:58,743 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:55:58,743 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:55:58,743 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:55:58,743 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:55:58,745 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:58,745 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:58,745 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:58,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:58,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:58,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:58,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:58,745 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:58,745 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:58,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:58,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:58,745 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:58,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:58,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:58,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:58,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:58,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:58,747 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:58,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:58,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:58,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:58,750 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:58,750 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:58,750 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:58,751 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:58,751 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:58,751 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:58,751 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:58,751 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:55:58,750 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:58,751 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:55:58,751 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:55:58,751 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:58,751 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:58,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:58,751 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:58,751 DEBUG [Thread-3183] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:55:58,751 DEBUG [Thread-3183] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:55:58,751 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:58,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:58,770 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp/C/a41059273f5d48348460c9f2aae7ed4a as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/C/a41059273f5d48348460c9f2aae7ed4a 2023-04-16 00:55:58,774 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in 0abd14932d4ec4e5fdc6177f2c7cc55c/C of 0abd14932d4ec4e5fdc6177f2c7cc55c into a41059273f5d48348460c9f2aae7ed4a(size=13.1 K), total size for store is 25.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 00:55:58,774 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0abd14932d4ec4e5fdc6177f2c7cc55c: 2023-04-16 00:55:58,774 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c., storeName=0abd14932d4ec4e5fdc6177f2c7cc55c/C, priority=9, startTime=1681606558269; duration=0sec 2023-04-16 00:55:58,774 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 00:55:58,786 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:58,786 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 327 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57268 deadline: 1681606618781, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:58,791 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=737 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp/A/40fc84d7e7b9466a866f046bdf99118c 2023-04-16 00:55:58,806 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=737 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp/B/c3058cdcd16644c99ff7ba70db719068 2023-04-16 00:55:58,822 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=737 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp/C/f753f0f3747a4467b70ea034356954c7 2023-04-16 00:55:58,826 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp/A/40fc84d7e7b9466a866f046bdf99118c as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/A/40fc84d7e7b9466a866f046bdf99118c 2023-04-16 00:55:58,829 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/A/40fc84d7e7b9466a866f046bdf99118c, entries=200, sequenceid=737, filesize=14.3 K 2023-04-16 00:55:58,830 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp/B/c3058cdcd16644c99ff7ba70db719068 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/B/c3058cdcd16644c99ff7ba70db719068 2023-04-16 00:55:58,833 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/B/c3058cdcd16644c99ff7ba70db719068, entries=150, sequenceid=737, filesize=11.9 K 2023-04-16 00:55:58,834 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp/C/f753f0f3747a4467b70ea034356954c7 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/C/f753f0f3747a4467b70ea034356954c7 2023-04-16 00:55:58,837 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/C/f753f0f3747a4467b70ea034356954c7, entries=150, sequenceid=737, filesize=11.9 K 2023-04-16 00:55:58,837 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for 0abd14932d4ec4e5fdc6177f2c7cc55c in 453ms, sequenceid=737, compaction requested=true 2023-04-16 00:55:58,838 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 0abd14932d4ec4e5fdc6177f2c7cc55c: 2023-04-16 00:55:58,838 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 00:55:58,838 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-16 00:55:58,838 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 00:55:58,838 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-16 00:55:58,838 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 00:55:58,839 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 42630 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-16 00:55:58,839 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HStore(1912): 0abd14932d4ec4e5fdc6177f2c7cc55c/A is initiating minor compaction (all files) 2023-04-16 00:55:58,839 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37750 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-16 00:55:58,839 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 0abd14932d4ec4e5fdc6177f2c7cc55c/A in TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. 2023-04-16 00:55:58,839 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HStore(1912): 0abd14932d4ec4e5fdc6177f2c7cc55c/B is initiating minor compaction (all files) 2023-04-16 00:55:58,839 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/A/e493d1145a3e4ffdb15b77bc9b9e00ec, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/A/dd7214cf35164c9e97fe1d6b6b2cf90c, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/A/40fc84d7e7b9466a866f046bdf99118c] into tmpdir=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp, totalSize=41.6 K 2023-04-16 00:55:58,839 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 0abd14932d4ec4e5fdc6177f2c7cc55c/B in TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. 2023-04-16 00:55:58,839 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/B/eac3fce8f98b4eaabf995b5dc9ecacb9, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/B/dbb5eea9c8dc47a79a432c684793be13, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/B/c3058cdcd16644c99ff7ba70db719068] into tmpdir=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp, totalSize=36.9 K 2023-04-16 00:55:58,840 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.Compactor(207): Compacting e493d1145a3e4ffdb15b77bc9b9e00ec, keycount=150, bloomtype=ROW, size=13.1 K, encoding=NONE, compression=NONE, seqNum=702, earliestPutTs=1681606558075 2023-04-16 00:55:58,840 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting eac3fce8f98b4eaabf995b5dc9ecacb9, keycount=150, bloomtype=ROW, size=13.1 K, encoding=NONE, compression=NONE, seqNum=702, earliestPutTs=1681606558075 2023-04-16 00:55:58,840 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.Compactor(207): Compacting dd7214cf35164c9e97fe1d6b6b2cf90c, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=716, earliestPutTs=1681606558251 2023-04-16 00:55:58,840 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting dbb5eea9c8dc47a79a432c684793be13, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=716, earliestPutTs=1681606558251 2023-04-16 00:55:58,840 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.Compactor(207): Compacting 40fc84d7e7b9466a866f046bdf99118c, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=737, earliestPutTs=1681606558344 2023-04-16 00:55:58,840 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting c3058cdcd16644c99ff7ba70db719068, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=737, earliestPutTs=1681606558344 2023-04-16 00:55:58,848 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] throttle.PressureAwareThroughputController(145): 0abd14932d4ec4e5fdc6177f2c7cc55c#B#compaction#519 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-16 00:55:58,851 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] throttle.PressureAwareThroughputController(145): 0abd14932d4ec4e5fdc6177f2c7cc55c#A#compaction#520 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-16 00:55:58,851 DEBUG [Thread-3183] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:55:58,852 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:55:58,854 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:55:58,856 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:55:58,857 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:55:58,857 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:55:58,857 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:58,857 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:55:58,857 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:55:58,857 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:58,857 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:58,858 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:58,859 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:58,859 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:55:58,860 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:58,860 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:58,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:58,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:58,861 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:58,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:55:58,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:58,861 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:55:58,861 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:55:58,862 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:55:58,862 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:55:58,862 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:58,862 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:55:58,862 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:55:58,862 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:55:58,862 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. 2023-04-16 00:55:58,862 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. started... 2023-04-16 00:55:58,863 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 0abd14932d4ec4e5fdc6177f2c7cc55c 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-16 00:55:58,863 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0abd14932d4ec4e5fdc6177f2c7cc55c, store=A 2023-04-16 00:55:58,863 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:55:58,863 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0abd14932d4ec4e5fdc6177f2c7cc55c, store=B 2023-04-16 00:55:58,863 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:55:58,863 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0abd14932d4ec4e5fdc6177f2c7cc55c, store=C 2023-04-16 00:55:58,863 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:55:58,864 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp/B/95dab1902c684fe287c95c641847484f as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/B/95dab1902c684fe287c95c641847484f 2023-04-16 00:55:58,868 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp/A/c1e14b4655e441bfacb8dd781b679359 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/A/c1e14b4655e441bfacb8dd781b679359 2023-04-16 00:55:58,870 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 0abd14932d4ec4e5fdc6177f2c7cc55c/B of 0abd14932d4ec4e5fdc6177f2c7cc55c into 95dab1902c684fe287c95c641847484f(size=13.2 K), total size for store is 13.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 00:55:58,870 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0abd14932d4ec4e5fdc6177f2c7cc55c: 2023-04-16 00:55:58,870 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c., storeName=0abd14932d4ec4e5fdc6177f2c7cc55c/B, priority=13, startTime=1681606558838; duration=0sec 2023-04-16 00:55:58,870 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 00:55:58,870 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-16 00:55:58,871 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37750 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-16 00:55:58,871 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HStore(1912): 0abd14932d4ec4e5fdc6177f2c7cc55c/C is initiating minor compaction (all files) 2023-04-16 00:55:58,871 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 0abd14932d4ec4e5fdc6177f2c7cc55c/C in TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. 2023-04-16 00:55:58,871 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/C/a41059273f5d48348460c9f2aae7ed4a, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/C/999b744968634309b19688352170de24, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/C/f753f0f3747a4467b70ea034356954c7] into tmpdir=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp, totalSize=36.9 K 2023-04-16 00:55:58,872 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting a41059273f5d48348460c9f2aae7ed4a, keycount=150, bloomtype=ROW, size=13.1 K, encoding=NONE, compression=NONE, seqNum=702, earliestPutTs=1681606558075 2023-04-16 00:55:58,872 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting 999b744968634309b19688352170de24, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=716, earliestPutTs=1681606558251 2023-04-16 00:55:58,873 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting f753f0f3747a4467b70ea034356954c7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=737, earliestPutTs=1681606558344 2023-04-16 00:55:58,879 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 0abd14932d4ec4e5fdc6177f2c7cc55c/A of 0abd14932d4ec4e5fdc6177f2c7cc55c into c1e14b4655e441bfacb8dd781b679359(size=13.2 K), total size for store is 13.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 00:55:58,879 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0abd14932d4ec4e5fdc6177f2c7cc55c: 2023-04-16 00:55:58,879 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c., storeName=0abd14932d4ec4e5fdc6177f2c7cc55c/A, priority=13, startTime=1681606558838; duration=0sec 2023-04-16 00:55:58,879 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 00:55:58,881 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] throttle.PressureAwareThroughputController(145): 0abd14932d4ec4e5fdc6177f2c7cc55c#C#compaction#522 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-16 00:55:58,886 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=751 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp/A/9ed538e97e95423f8ea10c10c7d87c63 2023-04-16 00:55:58,906 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp/C/ecf818a0d2c6456697925e8a409e692d as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/C/ecf818a0d2c6456697925e8a409e692d 2023-04-16 00:55:58,908 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=751 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp/B/4ba88f818e74441a82b2e3e1206d7295 2023-04-16 00:55:58,913 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 0abd14932d4ec4e5fdc6177f2c7cc55c/C of 0abd14932d4ec4e5fdc6177f2c7cc55c into ecf818a0d2c6456697925e8a409e692d(size=13.2 K), total size for store is 13.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 00:55:58,913 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0abd14932d4ec4e5fdc6177f2c7cc55c: 2023-04-16 00:55:58,913 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c., storeName=0abd14932d4ec4e5fdc6177f2c7cc55c/C, priority=13, startTime=1681606558838; duration=0sec 2023-04-16 00:55:58,913 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 00:55:58,941 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=751 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp/C/c395859588b643cc94e29780bf1d7651 2023-04-16 00:55:58,944 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp/A/9ed538e97e95423f8ea10c10c7d87c63 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/A/9ed538e97e95423f8ea10c10c7d87c63 2023-04-16 00:55:58,948 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/A/9ed538e97e95423f8ea10c10c7d87c63, entries=150, sequenceid=751, filesize=11.9 K 2023-04-16 00:55:58,950 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp/B/4ba88f818e74441a82b2e3e1206d7295 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/B/4ba88f818e74441a82b2e3e1206d7295 2023-04-16 00:55:58,953 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/B/4ba88f818e74441a82b2e3e1206d7295, entries=150, sequenceid=751, filesize=11.9 K 2023-04-16 00:55:58,954 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp/C/c395859588b643cc94e29780bf1d7651 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/C/c395859588b643cc94e29780bf1d7651 2023-04-16 00:55:58,957 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/C/c395859588b643cc94e29780bf1d7651, entries=150, sequenceid=751, filesize=11.9 K 2023-04-16 00:55:58,957 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=0 B/0 for 0abd14932d4ec4e5fdc6177f2c7cc55c in 95ms, sequenceid=751, compaction requested=false 2023-04-16 00:55:58,957 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 0abd14932d4ec4e5fdc6177f2c7cc55c: 2023-04-16 00:55:58,957 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. 2023-04-16 00:55:58,957 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:55:58,958 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:55:58,958 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:58,958 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:55:58,958 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:55:58,959 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:58,959 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:58,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:58,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:58,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:58,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:58,960 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:58,960 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:55:58,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:58,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:58,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:58,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:58,961 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:55:58,961 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2eaf5173[Count = 0] remaining members to acquire global barrier 2023-04-16 00:55:58,961 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:55:58,961 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:58,963 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:58,963 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:58,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:58,963 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:55:58,963 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:55:58,963 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:55:58,963 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:58,963 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:55:58,965 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:58,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:58,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:58,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:58,965 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:55:58,965 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:58,965 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:55:58,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:58,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:58,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:58,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:58,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:58,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:58,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:58,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:55:58,967 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:55:58,967 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:55:58,967 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:55:58,967 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:55:58,967 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:55:58,967 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:55:58,969 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:58,969 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:58,969 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:58,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:58,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:58,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:58,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:58,969 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:58,970 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:58,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:58,970 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:58,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:58,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:58,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:58,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:58,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:58,971 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:58,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:58,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:58,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:58,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:58,974 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:58,974 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:58,974 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:58,974 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:58,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:58,974 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:58,974 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:58,974 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:58,974 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:58,974 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:58,974 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:55:58,974 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:55:58,974 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:58,974 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:55:58,974 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:58,974 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:58,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:58,974 DEBUG [Thread-3183] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:55:58,974 DEBUG [Thread-3183] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:55:58,997 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(9158): Flush requested on 0abd14932d4ec4e5fdc6177f2c7cc55c 2023-04-16 00:55:58,997 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 0abd14932d4ec4e5fdc6177f2c7cc55c 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-16 00:55:58,997 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0abd14932d4ec4e5fdc6177f2c7cc55c, store=A 2023-04-16 00:55:58,997 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:55:58,997 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0abd14932d4ec4e5fdc6177f2c7cc55c, store=B 2023-04-16 00:55:58,997 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:55:58,997 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0abd14932d4ec4e5fdc6177f2c7cc55c, store=C 2023-04-16 00:55:58,997 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:55:59,003 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=764 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp/A/4db97e3ae479483a92bc130f4f04da2e 2023-04-16 00:55:59,011 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=764 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp/B/dee878d091d14cdb9e799fe52ed2d2f3 2023-04-16 00:55:59,074 DEBUG [Thread-3183] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:55:59,075 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:55:59,076 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:55:59,077 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:55:59,077 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:55:59,077 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:55:59,078 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:59,078 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:55:59,078 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:55:59,078 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:59,078 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:59,079 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:59,079 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:59,079 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:59,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:59,079 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:59,079 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:55:59,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:59,080 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:59,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:55:59,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:59,080 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:55:59,081 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:55:59,081 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:55:59,081 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:55:59,081 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:59,081 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:55:59,081 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:55:59,081 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:55:59,081 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. 2023-04-16 00:55:59,081 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. started... 2023-04-16 00:55:59,081 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. as already flushing 2023-04-16 00:55:59,081 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. 2023-04-16 00:55:59,081 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:55:59,082 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:55:59,082 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:59,082 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:55:59,082 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:55:59,084 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:59,084 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:59,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:59,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:59,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:59,085 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:59,085 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:55:59,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:59,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:59,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:59,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:59,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:59,086 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:55:59,086 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1df4ef63[Count = 0] remaining members to acquire global barrier 2023-04-16 00:55:59,086 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:55:59,086 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:59,087 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:59,087 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:59,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:59,087 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:55:59,087 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:55:59,087 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:59,087 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:55:59,087 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:55:59,089 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:55:59,089 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:59,089 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:59,089 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:55:59,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:59,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:59,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:59,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:59,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:59,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:59,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:59,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:59,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:59,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:59,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:55:59,091 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:55:59,091 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:55:59,091 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:55:59,091 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:55:59,091 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:55:59,091 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:55:59,093 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:59,093 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:59,093 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:59,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:59,093 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:59,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:59,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:59,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:59,093 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:59,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:59,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:59,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:59,094 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:59,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:59,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:59,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:59,095 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:59,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:59,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:59,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:59,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:59,098 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:59,098 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:59,098 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:59,098 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:59,098 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:59,098 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:59,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:59,098 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:59,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:59,100 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:59,100 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:55:59,100 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:55:59,100 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:59,101 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:59,101 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:59,101 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:59,101 DEBUG [Thread-3183] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:55:59,101 DEBUG [Thread-3183] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:55:59,109 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681606559032, max timestamp=1681606559106Num uniques 859; 2023-04-16 00:55:59,109 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-16 00:55:59,110 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681606559032, max timestamp=1681606559106Num uniques 859; 2023-04-16 00:55:59,110 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681606559032, max timestamp=1681606559106Num uniques 859; 2023-04-16 00:55:59,110 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-16 00:55:59,110 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-16 00:55:59,152 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:59,152 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 338 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57268 deadline: 1681606619145, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:59,158 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:59,158 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 168 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57254 deadline: 1681606619147, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:59,158 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:59,158 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 341 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57202 deadline: 1681606619152, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:59,171 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:59,171 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 180 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57212 deadline: 1681606619166, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:59,173 DEBUG [Thread-3181] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8210 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c., hostname=jenkins-hbase4.apache.org,34975,1681606419217, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor32.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-16 00:55:59,201 DEBUG [Thread-3183] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:55:59,201 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:55:59,203 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:55:59,204 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:55:59,204 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:55:59,204 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:55:59,204 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:59,204 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:55:59,204 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:55:59,204 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:59,204 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:59,205 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:59,205 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:59,205 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:59,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:59,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:59,206 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:59,206 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:55:59,206 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:59,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:55:59,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:59,206 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:55:59,207 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:55:59,207 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:55:59,207 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:55:59,207 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:59,207 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:55:59,207 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:55:59,207 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:55:59,207 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. 2023-04-16 00:55:59,207 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. started... 2023-04-16 00:55:59,207 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. as already flushing 2023-04-16 00:55:59,207 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. 2023-04-16 00:55:59,207 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:55:59,207 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:55:59,207 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:59,207 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:55:59,207 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:55:59,209 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:59,209 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:59,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:59,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:59,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:59,210 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:59,210 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:55:59,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:59,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:59,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:59,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:59,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:59,211 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:55:59,211 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6c539af4[Count = 0] remaining members to acquire global barrier 2023-04-16 00:55:59,211 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:55:59,211 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:59,212 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:59,212 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:59,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:59,212 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:55:59,212 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:55:59,212 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:55:59,212 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:59,212 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:55:59,222 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:59,222 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:55:59,222 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:59,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:59,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:59,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:59,222 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:55:59,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:59,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:59,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:59,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:59,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:59,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:59,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:59,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:55:59,224 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:55:59,224 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:55:59,224 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:55:59,224 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:55:59,224 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:55:59,224 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:55:59,225 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:59,226 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:59,226 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:59,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:59,226 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:59,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:59,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:59,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:59,226 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:59,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:59,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:59,226 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:59,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:59,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:59,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:59,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:59,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:59,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:59,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:59,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:59,228 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:59,231 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:59,231 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:59,231 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:59,231 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:59,231 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:59,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:59,231 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:59,231 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:59,231 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:55:59,231 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:55:59,231 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:59,231 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:59,231 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:59,231 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:59,231 DEBUG [Thread-3183] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:55:59,231 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:59,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:59,231 DEBUG [Thread-3183] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:55:59,259 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:59,259 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 340 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57268 deadline: 1681606619253, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:59,263 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:59,263 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 170 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57254 deadline: 1681606619259, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:59,263 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:59,263 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 343 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57202 deadline: 1681606619259, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:59,332 DEBUG [Thread-3183] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:55:59,332 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:55:59,334 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:55:59,334 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:55:59,335 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:55:59,335 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:55:59,335 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:59,335 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:55:59,335 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:55:59,335 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:59,335 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:59,336 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:59,336 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:59,336 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:59,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:59,337 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:59,337 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:55:59,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:59,337 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:59,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:55:59,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:59,337 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:55:59,338 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:55:59,338 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:55:59,338 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:55:59,338 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:59,338 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:55:59,338 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:55:59,338 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:55:59,338 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. 2023-04-16 00:55:59,338 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. started... 2023-04-16 00:55:59,338 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. as already flushing 2023-04-16 00:55:59,338 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. 2023-04-16 00:55:59,338 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:55:59,338 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:55:59,339 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:59,339 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:55:59,339 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:55:59,340 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:59,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:59,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:59,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:59,340 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:59,341 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:59,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:59,341 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:55:59,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:59,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:59,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:59,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:59,342 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:55:59,342 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@76c79f40[Count = 0] remaining members to acquire global barrier 2023-04-16 00:55:59,342 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:55:59,342 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:59,343 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:59,343 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:59,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:59,343 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:55:59,343 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:55:59,343 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:59,343 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:55:59,343 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:55:59,344 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:59,344 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:55:59,345 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:59,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:59,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:59,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:59,345 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:55:59,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:59,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:59,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:59,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:59,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:59,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:59,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:59,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:55:59,347 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:55:59,347 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:55:59,347 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:55:59,347 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:55:59,347 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:55:59,347 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:55:59,348 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:59,348 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:59,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:59,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:59,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:59,348 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:59,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:59,348 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:59,349 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:59,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:59,349 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:59,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:59,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:59,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:59,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:59,350 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:59,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:59,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:59,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:59,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:59,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:59,353 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:59,353 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:59,353 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:59,353 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:59,353 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:59,353 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:59,353 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:59,353 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:59,353 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:59,353 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:55:59,353 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:55:59,353 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:59,353 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:59,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:59,353 DEBUG [Thread-3183] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:55:59,353 DEBUG [Thread-3183] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:55:59,353 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:55:59,354 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:59,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:59,420 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=764 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp/C/0d0f26bb74ee45b0aab627371a4fcae5 2023-04-16 00:55:59,424 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp/A/4db97e3ae479483a92bc130f4f04da2e as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/A/4db97e3ae479483a92bc130f4f04da2e 2023-04-16 00:55:59,428 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/A/4db97e3ae479483a92bc130f4f04da2e, entries=150, sequenceid=764, filesize=11.9 K 2023-04-16 00:55:59,429 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp/B/dee878d091d14cdb9e799fe52ed2d2f3 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/B/dee878d091d14cdb9e799fe52ed2d2f3 2023-04-16 00:55:59,432 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/B/dee878d091d14cdb9e799fe52ed2d2f3, entries=150, sequenceid=764, filesize=11.9 K 2023-04-16 00:55:59,433 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp/C/0d0f26bb74ee45b0aab627371a4fcae5 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/C/0d0f26bb74ee45b0aab627371a4fcae5 2023-04-16 00:55:59,435 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/C/0d0f26bb74ee45b0aab627371a4fcae5, entries=150, sequenceid=764, filesize=11.9 K 2023-04-16 00:55:59,436 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 0abd14932d4ec4e5fdc6177f2c7cc55c in 439ms, sequenceid=764, compaction requested=true 2023-04-16 00:55:59,436 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 0abd14932d4ec4e5fdc6177f2c7cc55c: 2023-04-16 00:55:59,436 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 00:55:59,436 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 00:55:59,436 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-16 00:55:59,436 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-16 00:55:59,437 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-16 00:55:59,437 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37852 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-16 00:55:59,437 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37852 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-16 00:55:59,438 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HStore(1912): 0abd14932d4ec4e5fdc6177f2c7cc55c/A is initiating minor compaction (all files) 2023-04-16 00:55:59,438 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HStore(1912): 0abd14932d4ec4e5fdc6177f2c7cc55c/B is initiating minor compaction (all files) 2023-04-16 00:55:59,438 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 0abd14932d4ec4e5fdc6177f2c7cc55c/A in TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. 2023-04-16 00:55:59,438 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 0abd14932d4ec4e5fdc6177f2c7cc55c/B in TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. 2023-04-16 00:55:59,438 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/A/c1e14b4655e441bfacb8dd781b679359, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/A/9ed538e97e95423f8ea10c10c7d87c63, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/A/4db97e3ae479483a92bc130f4f04da2e] into tmpdir=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp, totalSize=37.0 K 2023-04-16 00:55:59,438 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/B/95dab1902c684fe287c95c641847484f, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/B/4ba88f818e74441a82b2e3e1206d7295, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/B/dee878d091d14cdb9e799fe52ed2d2f3] into tmpdir=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp, totalSize=37.0 K 2023-04-16 00:55:59,438 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.Compactor(207): Compacting c1e14b4655e441bfacb8dd781b679359, keycount=150, bloomtype=ROW, size=13.2 K, encoding=NONE, compression=NONE, seqNum=737, earliestPutTs=1681606558344 2023-04-16 00:55:59,438 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting 95dab1902c684fe287c95c641847484f, keycount=150, bloomtype=ROW, size=13.2 K, encoding=NONE, compression=NONE, seqNum=737, earliestPutTs=1681606558344 2023-04-16 00:55:59,439 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.Compactor(207): Compacting 9ed538e97e95423f8ea10c10c7d87c63, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=751, earliestPutTs=1681606558435 2023-04-16 00:55:59,439 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting 4ba88f818e74441a82b2e3e1206d7295, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=751, earliestPutTs=1681606558435 2023-04-16 00:55:59,439 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.Compactor(207): Compacting 4db97e3ae479483a92bc130f4f04da2e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=764, earliestPutTs=1681606558990 2023-04-16 00:55:59,439 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting dee878d091d14cdb9e799fe52ed2d2f3, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=764, earliestPutTs=1681606558990 2023-04-16 00:55:59,445 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] throttle.PressureAwareThroughputController(145): 0abd14932d4ec4e5fdc6177f2c7cc55c#A#compaction#528 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-16 00:55:59,445 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] throttle.PressureAwareThroughputController(145): 0abd14932d4ec4e5fdc6177f2c7cc55c#B#compaction#529 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-16 00:55:59,454 DEBUG [Thread-3183] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:55:59,454 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:55:59,456 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:55:59,460 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:55:59,460 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:55:59,460 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:55:59,460 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:59,460 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:55:59,460 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:55:59,460 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:59,460 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:59,462 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:59,462 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:59,462 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:59,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:59,462 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:59,462 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:55:59,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:59,463 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:59,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:55:59,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:59,464 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:55:59,464 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp/A/2e2bba7d0b8c44e59a06bfe2fb1ad054 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/A/2e2bba7d0b8c44e59a06bfe2fb1ad054 2023-04-16 00:55:59,464 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:55:59,464 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:55:59,464 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:55:59,464 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:55:59,464 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:55:59,465 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:55:59,465 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:55:59,465 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. 2023-04-16 00:55:59,465 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. started... 2023-04-16 00:55:59,465 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 0abd14932d4ec4e5fdc6177f2c7cc55c 3/3 column families, dataSize=134.18 KB heapSize=380.30 KB 2023-04-16 00:55:59,465 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0abd14932d4ec4e5fdc6177f2c7cc55c, store=A 2023-04-16 00:55:59,467 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 00:55:59,467 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0abd14932d4ec4e5fdc6177f2c7cc55c, store=B 2023-04-16 00:55:59,467 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 00:55:59,467 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0abd14932d4ec4e5fdc6177f2c7cc55c, store=C 2023-04-16 00:55:59,467 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 00:55:59,469 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(9158): Flush requested on 0abd14932d4ec4e5fdc6177f2c7cc55c 2023-04-16 00:55:59,469 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. as already flushing 2023-04-16 00:55:59,469 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 0abd14932d4ec4e5fdc6177f2c7cc55c/A of 0abd14932d4ec4e5fdc6177f2c7cc55c into 2e2bba7d0b8c44e59a06bfe2fb1ad054(size=13.3 K), total size for store is 13.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 00:55:59,469 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0abd14932d4ec4e5fdc6177f2c7cc55c: 2023-04-16 00:55:59,469 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c., storeName=0abd14932d4ec4e5fdc6177f2c7cc55c/A, priority=13, startTime=1681606559436; duration=0sec 2023-04-16 00:55:59,469 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 00:55:59,469 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-16 00:55:59,482 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37852 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-16 00:55:59,483 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HStore(1912): 0abd14932d4ec4e5fdc6177f2c7cc55c/C is initiating minor compaction (all files) 2023-04-16 00:55:59,483 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 0abd14932d4ec4e5fdc6177f2c7cc55c/C in TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. 2023-04-16 00:55:59,483 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/C/ecf818a0d2c6456697925e8a409e692d, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/C/c395859588b643cc94e29780bf1d7651, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/C/0d0f26bb74ee45b0aab627371a4fcae5] into tmpdir=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp, totalSize=37.0 K 2023-04-16 00:55:59,484 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.Compactor(207): Compacting ecf818a0d2c6456697925e8a409e692d, keycount=150, bloomtype=ROW, size=13.2 K, encoding=NONE, compression=NONE, seqNum=737, earliestPutTs=1681606558344 2023-04-16 00:55:59,484 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.Compactor(207): Compacting c395859588b643cc94e29780bf1d7651, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=751, earliestPutTs=1681606558435 2023-04-16 00:55:59,488 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.Compactor(207): Compacting 0d0f26bb74ee45b0aab627371a4fcae5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=764, earliestPutTs=1681606558990 2023-04-16 00:55:59,494 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] throttle.PressureAwareThroughputController(145): 0abd14932d4ec4e5fdc6177f2c7cc55c#C#compaction#531 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-16 00:55:59,500 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:59,500 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 344 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57268 deadline: 1681606619489, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:59,501 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:59,501 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 347 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57202 deadline: 1681606619489, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:59,504 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:59,504 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 175 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57254 deadline: 1681606619501, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:59,508 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=800 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp/A/29812f77ac024216be8f2520f5a9393e 2023-04-16 00:55:59,519 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=800 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp/B/86a8ab0f049e407c95f96d85c35b806f 2023-04-16 00:55:59,607 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:59,608 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 346 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57268 deadline: 1681606619601, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:59,608 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:59,608 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 349 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57202 deadline: 1681606619602, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:59,608 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:59,608 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 177 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57254 deadline: 1681606619605, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:59,813 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:59,813 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 351 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57202 deadline: 1681606619809, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:59,813 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:59,813 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 348 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57268 deadline: 1681606619810, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:59,814 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:55:59,814 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 179 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57254 deadline: 1681606619810, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:59,862 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp/B/bc18294df4f64767888c14fc4206eee1 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/B/bc18294df4f64767888c14fc4206eee1 2023-04-16 00:55:59,867 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 0abd14932d4ec4e5fdc6177f2c7cc55c/B of 0abd14932d4ec4e5fdc6177f2c7cc55c into bc18294df4f64767888c14fc4206eee1(size=13.3 K), total size for store is 13.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 00:55:59,868 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0abd14932d4ec4e5fdc6177f2c7cc55c: 2023-04-16 00:55:59,868 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c., storeName=0abd14932d4ec4e5fdc6177f2c7cc55c/B, priority=13, startTime=1681606559436; duration=0sec 2023-04-16 00:55:59,868 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 00:55:59,914 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp/C/8811c4ec139b412d86d2fda462c0c724 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/C/8811c4ec139b412d86d2fda462c0c724 2023-04-16 00:55:59,918 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 0abd14932d4ec4e5fdc6177f2c7cc55c/C of 0abd14932d4ec4e5fdc6177f2c7cc55c into 8811c4ec139b412d86d2fda462c0c724(size=13.3 K), total size for store is 13.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 00:55:59,918 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0abd14932d4ec4e5fdc6177f2c7cc55c: 2023-04-16 00:55:59,918 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c., storeName=0abd14932d4ec4e5fdc6177f2c7cc55c/C, priority=13, startTime=1681606559436; duration=0sec 2023-04-16 00:55:59,918 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 00:55:59,934 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=800 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp/C/f372d8cb7972482bacdf735400486e8a 2023-04-16 00:55:59,937 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp/A/29812f77ac024216be8f2520f5a9393e as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/A/29812f77ac024216be8f2520f5a9393e 2023-04-16 00:55:59,946 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/A/29812f77ac024216be8f2520f5a9393e, entries=150, sequenceid=800, filesize=11.9 K 2023-04-16 00:55:59,947 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp/B/86a8ab0f049e407c95f96d85c35b806f as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/B/86a8ab0f049e407c95f96d85c35b806f 2023-04-16 00:55:59,950 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/B/86a8ab0f049e407c95f96d85c35b806f, entries=150, sequenceid=800, filesize=11.9 K 2023-04-16 00:55:59,951 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp/C/f372d8cb7972482bacdf735400486e8a as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/C/f372d8cb7972482bacdf735400486e8a 2023-04-16 00:55:59,955 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/C/f372d8cb7972482bacdf735400486e8a, entries=150, sequenceid=800, filesize=11.9 K 2023-04-16 00:55:59,955 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~380.25 KB/389376, currentSize=46.96 KB/48090 for 0abd14932d4ec4e5fdc6177f2c7cc55c in 490ms, sequenceid=800, compaction requested=false 2023-04-16 00:55:59,955 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 0abd14932d4ec4e5fdc6177f2c7cc55c: 2023-04-16 00:55:59,955 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. 2023-04-16 00:55:59,955 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:55:59,956 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:55:59,956 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:59,956 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:55:59,956 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:55:59,957 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:59,957 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:59,957 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:59,958 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:55:59,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:59,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:59,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:59,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:59,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:59,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:59,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:59,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:59,960 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:55:59,960 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@22376c22[Count = 0] remaining members to acquire global barrier 2023-04-16 00:55:59,960 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:55:59,960 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:59,962 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:59,962 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:59,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:59,962 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:55:59,962 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:55:59,962 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:55:59,962 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:59,962 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:55:59,964 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:59,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:59,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:59,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:59,964 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:55:59,964 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:59,964 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:55:59,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:59,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:59,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:59,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:59,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:59,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:59,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:59,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:55:59,966 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:55:59,966 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:55:59,966 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:55:59,967 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:55:59,967 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:55:59,967 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:55:59,968 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:59,968 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:59,968 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:59,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:59,968 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:59,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:59,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:55:59,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:55:59,968 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:59,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:55:59,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:59,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:55:59,969 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:59,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:59,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:55:59,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:59,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:59,970 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:59,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:55:59,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:55:59,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:59,973 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:59,973 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:55:59,973 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:59,973 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:55:59,973 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:55:59,974 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:55:59,973 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:55:59,973 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:55:59,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:55:59,973 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:55:59,974 DEBUG [Thread-3183] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:55:59,974 DEBUG [Thread-3183] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:55:59,974 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:55:59,974 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:59,974 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:55:59,974 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:55:59,974 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:55:59,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:56:00,074 DEBUG [Thread-3183] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:56:00,075 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:56:00,076 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:56:00,078 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:56:00,078 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:56:00,078 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:56:00,078 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:56:00,078 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:56:00,078 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:56:00,078 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:00,078 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:00,080 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:56:00,080 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:56:00,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:56:00,080 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:00,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:00,080 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:00,080 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:56:00,080 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:00,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:56:00,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:00,081 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:56:00,081 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:56:00,081 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:56:00,081 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:56:00,081 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:56:00,081 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:56:00,081 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:56:00,081 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:56:00,081 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. 2023-04-16 00:56:00,082 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. started... 2023-04-16 00:56:00,083 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 0abd14932d4ec4e5fdc6177f2c7cc55c 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-16 00:56:00,083 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0abd14932d4ec4e5fdc6177f2c7cc55c, store=A 2023-04-16 00:56:00,083 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:56:00,083 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0abd14932d4ec4e5fdc6177f2c7cc55c, store=B 2023-04-16 00:56:00,083 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:56:00,084 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0abd14932d4ec4e5fdc6177f2c7cc55c, store=C 2023-04-16 00:56:00,084 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:56:00,119 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(9158): Flush requested on 0abd14932d4ec4e5fdc6177f2c7cc55c 2023-04-16 00:56:00,119 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. as already flushing 2023-04-16 00:56:00,159 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681606560149, max timestamp=1681606560158Num uniques 859; 2023-04-16 00:56:00,159 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-16 00:56:00,159 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681606560149, max timestamp=1681606560158Num uniques 859; 2023-04-16 00:56:00,159 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681606560149, max timestamp=1681606560158Num uniques 859; 2023-04-16 00:56:00,159 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-16 00:56:00,159 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-16 00:56:00,205 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:00,205 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 361 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57268 deadline: 1681606620200, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:00,211 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:00,211 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 192 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57254 deadline: 1681606620205, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:00,212 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:00,212 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 365 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57202 deadline: 1681606620205, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:00,311 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:00,311 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 363 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57268 deadline: 1681606620306, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:00,316 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:00,317 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 194 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57254 deadline: 1681606620312, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:00,318 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:00,318 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 367 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57202 deadline: 1681606620313, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:00,496 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=813 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp/A/5dae8985a1134867bb5a9dcf3e7603dd 2023-04-16 00:56:00,506 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=813 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp/B/2f93bdd985ba43ee9fa43d2d082d8481 2023-04-16 00:56:00,518 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=813 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp/C/585634914bd844f6adc60e42b14d1f61 2023-04-16 00:56:00,518 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:00,518 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 365 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57268 deadline: 1681606620513, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:00,523 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp/A/5dae8985a1134867bb5a9dcf3e7603dd as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/A/5dae8985a1134867bb5a9dcf3e7603dd 2023-04-16 00:56:00,524 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:00,524 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 196 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57254 deadline: 1681606620518, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:00,525 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:00,525 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 369 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57202 deadline: 1681606620520, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:00,527 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/A/5dae8985a1134867bb5a9dcf3e7603dd, entries=150, sequenceid=813, filesize=11.9 K 2023-04-16 00:56:00,528 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp/B/2f93bdd985ba43ee9fa43d2d082d8481 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/B/2f93bdd985ba43ee9fa43d2d082d8481 2023-04-16 00:56:00,532 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/B/2f93bdd985ba43ee9fa43d2d082d8481, entries=150, sequenceid=813, filesize=11.9 K 2023-04-16 00:56:00,533 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp/C/585634914bd844f6adc60e42b14d1f61 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/C/585634914bd844f6adc60e42b14d1f61 2023-04-16 00:56:00,537 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/C/585634914bd844f6adc60e42b14d1f61, entries=150, sequenceid=813, filesize=11.9 K 2023-04-16 00:56:00,538 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 0abd14932d4ec4e5fdc6177f2c7cc55c in 456ms, sequenceid=813, compaction requested=true 2023-04-16 00:56:00,538 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 0abd14932d4ec4e5fdc6177f2c7cc55c: 2023-04-16 00:56:00,538 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. 2023-04-16 00:56:00,538 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:56:00,538 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:56:00,538 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:00,538 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:56:00,538 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:56:00,540 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:00,540 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:00,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:00,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:00,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:00,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:00,542 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:00,542 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:56:00,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:00,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:00,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:00,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:00,543 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:56:00,543 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@188210f0[Count = 0] remaining members to acquire global barrier 2023-04-16 00:56:00,543 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:56:00,544 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:00,545 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:00,545 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:00,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:00,545 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:56:00,545 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:56:00,545 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:56:00,545 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:00,545 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:56:00,546 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:56:00,546 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:00,546 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:56:00,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:00,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:00,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:00,546 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:56:00,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:00,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:00,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:00,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:00,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:00,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:00,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:00,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:56:00,550 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:56:00,550 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:56:00,550 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:56:00,550 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:56:00,550 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:56:00,550 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:56:00,552 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:00,552 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:56:00,552 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:00,552 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:00,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:00,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:00,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:00,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:00,552 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:56:00,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:56:00,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:00,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:00,552 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:00,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:00,554 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:00,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:00,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:00,567 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:56:00,567 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:00,567 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:56:00,567 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:56:00,568 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:56:00,567 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:56:00,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:56:00,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:00,567 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:56:00,567 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:00,568 DEBUG [Thread-3183] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:56:00,568 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:00,568 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:00,568 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:56:00,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:56:00,568 DEBUG [Thread-3183] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:56:00,568 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:00,568 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:00,568 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:00,569 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:56:00,668 DEBUG [Thread-3183] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:56:00,669 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:56:00,670 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:56:00,671 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:56:00,672 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:56:00,672 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:56:00,672 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:56:00,672 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:56:00,672 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:56:00,672 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:00,672 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:00,673 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:56:00,674 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:56:00,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:56:00,674 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:00,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:00,674 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:00,674 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:56:00,674 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:00,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:56:00,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:00,675 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:56:00,675 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:56:00,675 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:56:00,675 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:56:00,675 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:56:00,675 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:56:00,675 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:56:00,675 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:56:00,675 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. 2023-04-16 00:56:00,675 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. started... 2023-04-16 00:56:00,676 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 0abd14932d4ec4e5fdc6177f2c7cc55c 3/3 column families, dataSize=134.18 KB heapSize=384.98 KB 2023-04-16 00:56:00,676 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0abd14932d4ec4e5fdc6177f2c7cc55c, store=A 2023-04-16 00:56:00,676 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 00:56:00,676 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0abd14932d4ec4e5fdc6177f2c7cc55c, store=B 2023-04-16 00:56:00,676 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 00:56:00,676 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0abd14932d4ec4e5fdc6177f2c7cc55c, store=C 2023-04-16 00:56:00,676 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 00:56:00,689 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=850 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp/A/6add4c0913b048998ff99cef9bfbc87f 2023-04-16 00:56:00,707 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=850 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp/B/36f805bec13542a78757d775f78a4d24 2023-04-16 00:56:00,717 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=850 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp/C/7130cada9d3f459dad331b68554df4ec 2023-04-16 00:56:00,720 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp/A/6add4c0913b048998ff99cef9bfbc87f as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/A/6add4c0913b048998ff99cef9bfbc87f 2023-04-16 00:56:00,723 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/A/6add4c0913b048998ff99cef9bfbc87f, entries=150, sequenceid=850, filesize=11.9 K 2023-04-16 00:56:00,723 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp/B/36f805bec13542a78757d775f78a4d24 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/B/36f805bec13542a78757d775f78a4d24 2023-04-16 00:56:00,726 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/B/36f805bec13542a78757d775f78a4d24, entries=150, sequenceid=850, filesize=11.9 K 2023-04-16 00:56:00,727 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp/C/7130cada9d3f459dad331b68554df4ec as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/C/7130cada9d3f459dad331b68554df4ec 2023-04-16 00:56:00,730 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/C/7130cada9d3f459dad331b68554df4ec, entries=150, sequenceid=850, filesize=11.9 K 2023-04-16 00:56:00,730 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~384.94 KB/394176, currentSize=0 B/0 for 0abd14932d4ec4e5fdc6177f2c7cc55c in 54ms, sequenceid=850, compaction requested=true 2023-04-16 00:56:00,730 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 0abd14932d4ec4e5fdc6177f2c7cc55c: 2023-04-16 00:56:00,730 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. 2023-04-16 00:56:00,730 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:56:00,731 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:56:00,731 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:00,731 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:56:00,731 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:56:00,732 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:00,732 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:00,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:00,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:00,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:00,733 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:00,733 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:56:00,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:00,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:00,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:00,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:00,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:00,734 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:56:00,734 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@68f6ad2e[Count = 0] remaining members to acquire global barrier 2023-04-16 00:56:00,734 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:56:00,735 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:00,736 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:00,736 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:00,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:00,736 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:00,736 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:56:00,736 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:56:00,736 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:56:00,736 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:56:00,739 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:56:00,739 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:56:00,739 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:56:00,740 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:00,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:00,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:00,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:00,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:00,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:00,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:00,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:00,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:00,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:00,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:00,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:56:00,742 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:56:00,742 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:56:00,742 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:56:00,742 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:56:00,742 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:56:00,742 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:56:00,744 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:00,744 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:56:00,744 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:00,744 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:00,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:00,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:00,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:00,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:00,744 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:56:00,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:56:00,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:00,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:00,745 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:00,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:00,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:00,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:00,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:00,747 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:00,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:00,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:00,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:00,750 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:56:00,750 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:56:00,750 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:00,750 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:00,750 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:00,750 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:00,750 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:00,750 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:56:00,750 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:56:00,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:56:00,750 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:56:00,750 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:56:00,750 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:00,750 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:00,750 DEBUG [Thread-3183] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:56:00,750 DEBUG [Thread-3183] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:56:00,751 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:56:00,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:56:00,841 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(9158): Flush requested on 0abd14932d4ec4e5fdc6177f2c7cc55c 2023-04-16 00:56:00,841 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 0abd14932d4ec4e5fdc6177f2c7cc55c 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-16 00:56:00,842 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0abd14932d4ec4e5fdc6177f2c7cc55c, store=A 2023-04-16 00:56:00,842 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:56:00,842 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0abd14932d4ec4e5fdc6177f2c7cc55c, store=B 2023-04-16 00:56:00,842 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:56:00,842 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0abd14932d4ec4e5fdc6177f2c7cc55c, store=C 2023-04-16 00:56:00,843 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:56:00,851 DEBUG [Thread-3183] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:56:00,851 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:56:00,856 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:56:00,857 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:56:00,857 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:56:00,858 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:56:00,858 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:56:00,858 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:56:00,858 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:56:00,858 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:00,858 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:00,859 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:56:00,859 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:00,859 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:56:00,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:56:00,860 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:00,860 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:56:00,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:00,860 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:00,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:56:00,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:00,861 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:56:00,861 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:56:00,861 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:56:00,861 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:56:00,861 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:56:00,861 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:56:00,861 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=861 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp/A/d9acb3e4c4994d8c976105a6d11ae91b 2023-04-16 00:56:00,861 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:56:00,861 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:56:00,861 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. 2023-04-16 00:56:00,861 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. started... 2023-04-16 00:56:00,862 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. as already flushing 2023-04-16 00:56:00,862 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. 2023-04-16 00:56:00,862 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:56:00,862 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:56:00,862 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:00,862 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:56:00,862 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:56:00,864 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:00,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:00,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:00,864 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:00,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:00,864 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:00,864 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:56:00,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:00,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:00,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:00,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:00,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:00,866 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:56:00,867 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@220112ef[Count = 0] remaining members to acquire global barrier 2023-04-16 00:56:00,867 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:56:00,867 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:00,868 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:00,868 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:00,868 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:56:00,869 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:00,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:00,869 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:56:00,869 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:56:00,869 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:56:00,870 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:00,870 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:56:00,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:00,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:00,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:00,870 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:56:00,870 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:56:00,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:00,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:00,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:00,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:00,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:00,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:00,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:00,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:56:00,873 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:56:00,873 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:56:00,873 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:56:00,873 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:56:00,873 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:56:00,873 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:56:00,875 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:00,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:00,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:00,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:00,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:00,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:00,876 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:00,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:00,877 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:00,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:00,876 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:56:00,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:00,878 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:56:00,878 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:00,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:56:00,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:00,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:00,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:00,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:00,879 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:00,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:00,882 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:00,882 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:56:00,882 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:00,882 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:56:00,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:56:00,882 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:00,882 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:56:00,882 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:56:00,882 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:56:00,882 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:56:00,882 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:00,883 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:00,883 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:00,883 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:00,883 DEBUG [Thread-3183] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:56:00,883 DEBUG [Thread-3183] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:56:00,883 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:56:00,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:56:00,907 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:00,907 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 377 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57268 deadline: 1681606620896, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:00,907 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:00,908 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 206 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57254 deadline: 1681606620904, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:00,908 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681606560888, max timestamp=1681606560906Num uniques 859; 2023-04-16 00:56:00,908 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-16 00:56:00,908 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681606560888, max timestamp=1681606560906Num uniques 859; 2023-04-16 00:56:00,908 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-16 00:56:00,909 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681606560888, max timestamp=1681606560906Num uniques 859; 2023-04-16 00:56:00,909 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-16 00:56:00,983 DEBUG [Thread-3183] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:56:00,983 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:56:00,985 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:56:00,986 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:56:00,986 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:56:00,986 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:56:00,986 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:56:00,986 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:56:00,986 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:56:00,986 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:00,986 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:00,987 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:56:00,987 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:56:00,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:56:00,987 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:00,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:00,988 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:00,988 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:56:00,988 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:00,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:56:00,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:00,988 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:56:00,989 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:56:00,989 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:56:00,989 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:56:00,989 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:56:00,989 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:56:00,989 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:56:00,989 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:56:00,989 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. 2023-04-16 00:56:00,989 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. started... 2023-04-16 00:56:00,989 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. as already flushing 2023-04-16 00:56:00,989 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. 2023-04-16 00:56:00,989 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:56:00,989 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:56:00,989 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:00,989 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:56:00,990 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:56:00,991 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:00,991 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:00,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:00,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:00,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:00,991 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:00,991 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:56:00,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:00,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:00,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:00,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:00,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:00,992 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:56:00,992 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3bce3cf1[Count = 0] remaining members to acquire global barrier 2023-04-16 00:56:00,992 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:56:00,992 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:00,993 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:00,994 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:00,994 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:56:00,994 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:00,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:00,994 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:56:00,994 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:56:00,994 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:56:00,996 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:56:00,996 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:56:00,996 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:00,996 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:56:00,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:00,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:00,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:00,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:00,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:00,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:00,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:00,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:00,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:00,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:00,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:56:00,999 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:56:00,999 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:56:00,999 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:56:00,999 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:56:00,999 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:56:00,999 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:56:01,000 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:01,000 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:01,000 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:56:01,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:01,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:01,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:01,000 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:01,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:01,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:01,001 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:01,001 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:56:01,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:56:01,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:01,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:01,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:01,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:01,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:01,002 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:01,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:01,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:01,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:01,007 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:56:01,007 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:01,007 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:56:01,007 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:56:01,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:56:01,007 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:56:01,007 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:56:01,007 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:56:01,007 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:01,008 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:01,008 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:01,008 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:01,008 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:01,008 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:56:01,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:56:01,008 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:01,008 DEBUG [Thread-3183] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:56:01,008 DEBUG [Thread-3183] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:56:01,018 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:01,018 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 208 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57254 deadline: 1681606621010, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:01,019 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:01,019 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 380 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57268 deadline: 1681606621011, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:01,019 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:01,019 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 392 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57202 deadline: 1681606621018, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:01,108 DEBUG [Thread-3183] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:56:01,109 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:56:01,110 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:56:01,111 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:56:01,111 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:56:01,111 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:56:01,112 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:56:01,112 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:56:01,112 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:56:01,112 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:01,112 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:01,114 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:01,114 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:56:01,114 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:56:01,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:56:01,114 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:01,114 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:56:01,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:01,115 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:01,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:56:01,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:01,115 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:56:01,116 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:56:01,116 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:56:01,116 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:56:01,116 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:56:01,116 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:56:01,116 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:56:01,116 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:56:01,116 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. 2023-04-16 00:56:01,117 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. started... 2023-04-16 00:56:01,117 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. as already flushing 2023-04-16 00:56:01,117 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. 2023-04-16 00:56:01,117 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:56:01,117 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:56:01,117 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:01,117 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:56:01,117 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:56:01,118 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:01,118 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:01,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:01,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:01,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:01,119 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:01,119 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:56:01,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:01,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:01,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:01,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:01,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:01,120 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:56:01,120 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@33c3bee8[Count = 0] remaining members to acquire global barrier 2023-04-16 00:56:01,120 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:56:01,121 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:01,123 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:01,123 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:01,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:01,123 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:56:01,123 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:56:01,123 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:56:01,123 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:01,123 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:56:01,124 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:01,124 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:56:01,124 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:56:01,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:01,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:01,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:01,124 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:56:01,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:01,125 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:01,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:01,125 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 382 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57268 deadline: 1681606621119, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:01,125 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:01,125 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 394 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57202 deadline: 1681606621120, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:01,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:01,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:01,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:01,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:01,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:01,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:56:01,127 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:56:01,127 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:56:01,127 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:56:01,128 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:56:01,128 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:56:01,128 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:56:01,130 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:01,130 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:01,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:01,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:01,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:01,130 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:56:01,131 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:01,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:01,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:01,133 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:01,133 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:56:01,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:56:01,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:01,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:01,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:01,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:01,140 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:01,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:01,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:01,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:01,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:01,143 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:01,143 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:01,143 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:01,143 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:01,143 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:01,143 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:01,143 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:56:01,143 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:56:01,143 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:56:01,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:56:01,143 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:56:01,143 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:56:01,143 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:01,143 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:56:01,143 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:56:01,143 DEBUG [Thread-3183] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:56:01,144 DEBUG [Thread-3183] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:56:01,144 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:56:01,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:56:01,224 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:01,224 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 210 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57254 deadline: 1681606621220, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:01,244 DEBUG [Thread-3183] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:56:01,244 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:56:01,246 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:56:01,246 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:56:01,247 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:56:01,247 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:56:01,247 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:56:01,247 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:56:01,247 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:56:01,247 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:01,247 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:01,248 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:01,248 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:56:01,248 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:56:01,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:56:01,248 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:01,249 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:56:01,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:01,249 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:01,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:56:01,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:01,249 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:56:01,249 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:56:01,249 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:56:01,249 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:56:01,250 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:56:01,250 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:56:01,250 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:56:01,250 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:56:01,250 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. 2023-04-16 00:56:01,250 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. started... 2023-04-16 00:56:01,250 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. as already flushing 2023-04-16 00:56:01,250 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. 2023-04-16 00:56:01,250 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:56:01,250 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:56:01,250 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:01,250 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:56:01,250 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:56:01,251 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:01,252 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:01,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:01,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:01,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:01,252 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:01,252 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:56:01,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:01,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:01,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:01,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:01,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:01,253 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:56:01,253 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6e01ea63[Count = 0] remaining members to acquire global barrier 2023-04-16 00:56:01,253 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:56:01,253 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:01,256 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:01,256 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:01,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:01,256 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:56:01,256 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:56:01,256 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:56:01,257 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:01,257 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:56:01,258 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:01,258 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:56:01,258 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:56:01,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:01,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:01,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:01,258 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:56:01,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:01,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:01,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:01,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:01,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:01,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:01,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:01,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:56:01,260 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:56:01,260 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:56:01,260 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:56:01,260 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:56:01,260 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:56:01,260 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:56:01,262 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:01,262 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:01,262 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:01,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:01,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:01,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:01,262 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:56:01,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:01,263 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:01,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:01,263 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:56:01,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:56:01,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:01,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:01,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:01,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:01,264 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:01,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:01,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:01,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:01,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:01,265 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:01,265 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:01,265 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:01,265 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:56:01,265 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:01,266 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:01,266 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:01,266 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:01,266 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:56:01,266 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:56:01,266 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:56:01,266 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:56:01,266 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:56:01,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:56:01,266 DEBUG [Thread-3183] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:56:01,266 DEBUG [Thread-3183] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:56:01,266 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:56:01,266 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:56:01,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:56:01,277 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=861 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp/B/896454a8e74247afa37156e1830ad25e 2023-04-16 00:56:01,288 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=861 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp/C/4ddac710cbe442149c9b1d17a929ead2 2023-04-16 00:56:01,292 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp/A/d9acb3e4c4994d8c976105a6d11ae91b as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/A/d9acb3e4c4994d8c976105a6d11ae91b 2023-04-16 00:56:01,296 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/A/d9acb3e4c4994d8c976105a6d11ae91b, entries=200, sequenceid=861, filesize=14.3 K 2023-04-16 00:56:01,297 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp/B/896454a8e74247afa37156e1830ad25e as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/B/896454a8e74247afa37156e1830ad25e 2023-04-16 00:56:01,300 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/B/896454a8e74247afa37156e1830ad25e, entries=150, sequenceid=861, filesize=11.9 K 2023-04-16 00:56:01,300 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp/C/4ddac710cbe442149c9b1d17a929ead2 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/C/4ddac710cbe442149c9b1d17a929ead2 2023-04-16 00:56:01,303 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/C/4ddac710cbe442149c9b1d17a929ead2, entries=150, sequenceid=861, filesize=11.9 K 2023-04-16 00:56:01,303 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 0abd14932d4ec4e5fdc6177f2c7cc55c in 462ms, sequenceid=861, compaction requested=true 2023-04-16 00:56:01,304 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 0abd14932d4ec4e5fdc6177f2c7cc55c: 2023-04-16 00:56:01,304 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 00:56:01,304 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-16 00:56:01,304 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 00:56:01,304 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-16 00:56:01,304 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 00:56:01,307 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 64766 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-16 00:56:01,307 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 62326 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-16 00:56:01,307 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HStore(1912): 0abd14932d4ec4e5fdc6177f2c7cc55c/A is initiating minor compaction (all files) 2023-04-16 00:56:01,307 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HStore(1912): 0abd14932d4ec4e5fdc6177f2c7cc55c/B is initiating minor compaction (all files) 2023-04-16 00:56:01,307 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 0abd14932d4ec4e5fdc6177f2c7cc55c/A in TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. 2023-04-16 00:56:01,307 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 0abd14932d4ec4e5fdc6177f2c7cc55c/B in TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. 2023-04-16 00:56:01,307 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/A/2e2bba7d0b8c44e59a06bfe2fb1ad054, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/A/29812f77ac024216be8f2520f5a9393e, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/A/5dae8985a1134867bb5a9dcf3e7603dd, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/A/6add4c0913b048998ff99cef9bfbc87f, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/A/d9acb3e4c4994d8c976105a6d11ae91b] into tmpdir=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp, totalSize=63.2 K 2023-04-16 00:56:01,307 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/B/bc18294df4f64767888c14fc4206eee1, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/B/86a8ab0f049e407c95f96d85c35b806f, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/B/2f93bdd985ba43ee9fa43d2d082d8481, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/B/36f805bec13542a78757d775f78a4d24, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/B/896454a8e74247afa37156e1830ad25e] into tmpdir=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp, totalSize=60.9 K 2023-04-16 00:56:01,308 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.Compactor(207): Compacting 2e2bba7d0b8c44e59a06bfe2fb1ad054, keycount=150, bloomtype=ROW, size=13.3 K, encoding=NONE, compression=NONE, seqNum=764, earliestPutTs=1681606558990 2023-04-16 00:56:01,308 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting bc18294df4f64767888c14fc4206eee1, keycount=150, bloomtype=ROW, size=13.3 K, encoding=NONE, compression=NONE, seqNum=764, earliestPutTs=1681606558990 2023-04-16 00:56:01,308 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.Compactor(207): Compacting 29812f77ac024216be8f2520f5a9393e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=800, earliestPutTs=1681606559133 2023-04-16 00:56:01,308 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting 86a8ab0f049e407c95f96d85c35b806f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=800, earliestPutTs=1681606559133 2023-04-16 00:56:01,309 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.Compactor(207): Compacting 5dae8985a1134867bb5a9dcf3e7603dd, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=813, earliestPutTs=1681606559469 2023-04-16 00:56:01,309 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting 2f93bdd985ba43ee9fa43d2d082d8481, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=813, earliestPutTs=1681606559469 2023-04-16 00:56:01,309 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.Compactor(207): Compacting 6add4c0913b048998ff99cef9bfbc87f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=850, earliestPutTs=1681606560189 2023-04-16 00:56:01,309 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting 36f805bec13542a78757d775f78a4d24, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=850, earliestPutTs=1681606560189 2023-04-16 00:56:01,309 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.Compactor(207): Compacting d9acb3e4c4994d8c976105a6d11ae91b, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=861, earliestPutTs=1681606560826 2023-04-16 00:56:01,309 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting 896454a8e74247afa37156e1830ad25e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=861, earliestPutTs=1681606560826 2023-04-16 00:56:01,317 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] throttle.PressureAwareThroughputController(145): 0abd14932d4ec4e5fdc6177f2c7cc55c#B#compaction#544 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-16 00:56:01,318 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] throttle.PressureAwareThroughputController(145): 0abd14932d4ec4e5fdc6177f2c7cc55c#A#compaction#543 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-16 00:56:01,331 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(9158): Flush requested on 0abd14932d4ec4e5fdc6177f2c7cc55c 2023-04-16 00:56:01,331 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 0abd14932d4ec4e5fdc6177f2c7cc55c 3/3 column families, dataSize=134.18 KB heapSize=380.30 KB 2023-04-16 00:56:01,331 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0abd14932d4ec4e5fdc6177f2c7cc55c, store=A 2023-04-16 00:56:01,331 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 00:56:01,331 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0abd14932d4ec4e5fdc6177f2c7cc55c, store=B 2023-04-16 00:56:01,331 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 00:56:01,331 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0abd14932d4ec4e5fdc6177f2c7cc55c, store=C 2023-04-16 00:56:01,331 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 00:56:01,338 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp/B/8f2661f0a2654b90b609ea21bb1144ef as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/B/8f2661f0a2654b90b609ea21bb1144ef 2023-04-16 00:56:01,343 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 0abd14932d4ec4e5fdc6177f2c7cc55c/B of 0abd14932d4ec4e5fdc6177f2c7cc55c into 8f2661f0a2654b90b609ea21bb1144ef(size=13.4 K), total size for store is 13.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 00:56:01,343 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0abd14932d4ec4e5fdc6177f2c7cc55c: 2023-04-16 00:56:01,344 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c., storeName=0abd14932d4ec4e5fdc6177f2c7cc55c/B, priority=11, startTime=1681606561304; duration=0sec 2023-04-16 00:56:01,344 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 00:56:01,344 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-16 00:56:01,345 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 62326 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-16 00:56:01,345 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HStore(1912): 0abd14932d4ec4e5fdc6177f2c7cc55c/C is initiating minor compaction (all files) 2023-04-16 00:56:01,345 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 0abd14932d4ec4e5fdc6177f2c7cc55c/C in TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. 2023-04-16 00:56:01,346 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/C/8811c4ec139b412d86d2fda462c0c724, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/C/f372d8cb7972482bacdf735400486e8a, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/C/585634914bd844f6adc60e42b14d1f61, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/C/7130cada9d3f459dad331b68554df4ec, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/C/4ddac710cbe442149c9b1d17a929ead2] into tmpdir=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp, totalSize=60.9 K 2023-04-16 00:56:01,346 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting 8811c4ec139b412d86d2fda462c0c724, keycount=150, bloomtype=ROW, size=13.3 K, encoding=NONE, compression=NONE, seqNum=764, earliestPutTs=1681606558990 2023-04-16 00:56:01,346 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting f372d8cb7972482bacdf735400486e8a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=800, earliestPutTs=1681606559133 2023-04-16 00:56:01,346 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting 585634914bd844f6adc60e42b14d1f61, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=813, earliestPutTs=1681606559469 2023-04-16 00:56:01,347 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting 7130cada9d3f459dad331b68554df4ec, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=850, earliestPutTs=1681606560189 2023-04-16 00:56:01,347 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting 4ddac710cbe442149c9b1d17a929ead2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=861, earliestPutTs=1681606560826 2023-04-16 00:56:01,356 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp/A/f43ab8c331144c7e97553c84937b8941 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/A/f43ab8c331144c7e97553c84937b8941 2023-04-16 00:56:01,357 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] throttle.PressureAwareThroughputController(145): 0abd14932d4ec4e5fdc6177f2c7cc55c#C#compaction#546 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-16 00:56:01,366 DEBUG [Thread-3183] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:56:01,367 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:56:01,372 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:56:01,373 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 0abd14932d4ec4e5fdc6177f2c7cc55c/A of 0abd14932d4ec4e5fdc6177f2c7cc55c into f43ab8c331144c7e97553c84937b8941(size=13.4 K), total size for store is 13.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 00:56:01,373 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0abd14932d4ec4e5fdc6177f2c7cc55c: 2023-04-16 00:56:01,373 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c., storeName=0abd14932d4ec4e5fdc6177f2c7cc55c/A, priority=11, startTime=1681606561304; duration=0sec 2023-04-16 00:56:01,373 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 00:56:01,377 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:56:01,377 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:56:01,377 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:56:01,377 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:56:01,377 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:56:01,377 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:56:01,377 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:01,378 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:01,379 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:56:01,379 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:01,379 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:56:01,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:56:01,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:01,380 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:01,380 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:56:01,380 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:01,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:56:01,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:01,381 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:56:01,381 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp/C/0cb89dc6fcda43c8bdedb5d616e65663 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/C/0cb89dc6fcda43c8bdedb5d616e65663 2023-04-16 00:56:01,381 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:56:01,381 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:56:01,382 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:56:01,382 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:56:01,382 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:56:01,384 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:56:01,384 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:56:01,384 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. 2023-04-16 00:56:01,384 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. started... 2023-04-16 00:56:01,384 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. as already flushing 2023-04-16 00:56:01,384 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. 2023-04-16 00:56:01,384 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:56:01,384 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:56:01,384 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:01,384 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:56:01,384 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:56:01,385 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:01,385 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 400 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57202 deadline: 1681606621374, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:01,385 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:01,385 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 388 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57268 deadline: 1681606621377, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:01,385 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 0abd14932d4ec4e5fdc6177f2c7cc55c/C of 0abd14932d4ec4e5fdc6177f2c7cc55c into 0cb89dc6fcda43c8bdedb5d616e65663(size=13.4 K), total size for store is 13.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 00:56:01,385 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0abd14932d4ec4e5fdc6177f2c7cc55c: 2023-04-16 00:56:01,385 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c., storeName=0abd14932d4ec4e5fdc6177f2c7cc55c/C, priority=11, startTime=1681606561304; duration=0sec 2023-04-16 00:56:01,386 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:01,386 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:01,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:01,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:01,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:01,386 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 00:56:01,386 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:01,386 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:56:01,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:01,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:01,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:01,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:01,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:01,387 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:56:01,387 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@44bd0b26[Count = 0] remaining members to acquire global barrier 2023-04-16 00:56:01,387 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:56:01,388 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:01,389 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:01,389 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:01,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:01,389 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:56:01,389 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:56:01,389 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:56:01,389 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:01,389 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:56:01,390 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:56:01,390 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:01,390 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:56:01,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:01,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:01,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:01,390 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:56:01,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:01,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:01,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:01,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:01,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:01,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:01,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:01,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:56:01,392 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:56:01,392 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:56:01,392 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:56:01,392 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:56:01,392 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:56:01,392 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:56:01,395 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:01,395 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:56:01,395 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:01,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:01,395 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:01,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:01,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:01,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:01,395 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:56:01,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:56:01,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:01,395 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:01,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:01,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:01,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:01,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:01,396 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:01,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:01,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:01,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:01,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:01,398 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:01,398 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:01,398 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:01,398 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:01,398 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:56:01,398 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:01,398 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:56:01,398 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:01,398 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:56:01,398 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:56:01,398 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:01,398 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:56:01,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:56:01,399 DEBUG [Thread-3183] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:56:01,398 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:56:01,399 DEBUG [Thread-3183] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:56:01,399 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:56:01,399 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:56:01,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:56:01,488 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:01,489 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 402 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57202 deadline: 1681606621486, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:01,490 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:01,490 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 390 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57268 deadline: 1681606621486, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:01,499 DEBUG [Thread-3183] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:56:01,499 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:56:01,501 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:56:01,502 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:56:01,502 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:56:01,502 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:56:01,502 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:56:01,502 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:56:01,502 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:56:01,503 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:01,503 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:01,507 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:56:01,507 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:01,507 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:56:01,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:56:01,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:01,507 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:01,507 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:56:01,507 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:01,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:56:01,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:01,508 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:56:01,508 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:56:01,508 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:56:01,508 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:56:01,508 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:56:01,508 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:56:01,510 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:56:01,510 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:56:01,510 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. 2023-04-16 00:56:01,510 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. started... 2023-04-16 00:56:01,510 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. as already flushing 2023-04-16 00:56:01,510 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. 2023-04-16 00:56:01,510 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:56:01,510 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:56:01,510 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:01,510 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:56:01,510 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:56:01,513 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:01,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:01,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:01,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:01,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:01,514 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:01,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:01,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:01,515 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:01,515 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:56:01,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:01,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:01,516 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:56:01,517 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@e5e8ef4[Count = 0] remaining members to acquire global barrier 2023-04-16 00:56:01,517 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:56:01,517 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:01,518 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:01,518 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:01,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:01,518 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:56:01,518 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:56:01,518 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:56:01,519 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:01,519 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:56:01,521 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:56:01,521 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:56:01,521 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:01,521 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:56:01,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:01,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:01,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:01,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:01,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:01,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:01,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:01,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:01,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:01,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:01,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:56:01,524 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:56:01,524 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:56:01,525 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:56:01,525 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:56:01,525 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:56:01,525 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:56:01,526 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:01,526 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:01,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:01,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:01,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:01,526 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:01,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:01,526 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:56:01,527 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:01,527 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:56:01,527 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 212 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57254 deadline: 1681606621525, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:01,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:56:01,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:01,528 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:01,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:01,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:01,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:01,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:01,529 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:01,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:01,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:01,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:01,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:01,532 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:56:01,532 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:56:01,532 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:01,532 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:56:01,532 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:56:01,532 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:56:01,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:56:01,532 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:01,533 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:01,533 DEBUG [Thread-3183] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:56:01,533 DEBUG [Thread-3183] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:56:01,532 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:56:01,533 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:01,533 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:01,533 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:56:01,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:56:01,533 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:56:01,533 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:01,533 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:01,633 DEBUG [Thread-3183] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:56:01,633 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:56:01,635 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:56:01,636 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:56:01,636 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:56:01,636 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:56:01,636 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:56:01,636 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:56:01,636 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:56:01,637 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:01,637 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:01,638 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:01,638 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:56:01,638 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:56:01,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:56:01,638 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:01,638 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:56:01,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:01,639 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:01,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:56:01,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:01,639 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:56:01,639 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:56:01,639 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:56:01,639 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:56:01,639 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:56:01,639 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:56:01,639 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:56:01,639 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:56:01,639 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. 2023-04-16 00:56:01,640 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. started... 2023-04-16 00:56:01,640 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. as already flushing 2023-04-16 00:56:01,640 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. 2023-04-16 00:56:01,640 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:56:01,640 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:56:01,640 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:01,640 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:56:01,640 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:56:01,641 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:01,641 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:01,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:01,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:01,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:01,642 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:01,642 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:56:01,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:01,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:01,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:01,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:01,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:01,644 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:56:01,644 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@20c44529[Count = 0] remaining members to acquire global barrier 2023-04-16 00:56:01,644 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:56:01,644 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:01,645 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:01,645 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:01,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:01,645 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:56:01,645 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:56:01,645 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:56:01,646 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:01,646 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:56:01,648 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:01,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:01,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:01,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:01,648 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:56:01,648 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:56:01,649 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:56:01,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:01,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:01,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:01,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:01,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:01,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:01,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:01,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:56:01,651 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:56:01,651 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:56:01,651 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:56:01,651 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:56:01,651 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:56:01,651 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:56:01,653 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:01,653 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:01,653 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:01,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:01,653 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:56:01,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:01,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:01,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:01,653 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:56:01,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:56:01,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:01,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:01,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:01,654 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:01,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:01,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:01,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:01,655 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:01,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:01,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:01,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:01,657 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:01,657 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:01,657 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:01,657 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:56:01,657 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:56:01,657 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:01,657 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:01,657 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:01,657 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:01,657 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:56:01,657 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:56:01,657 DEBUG [Thread-3183] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:56:01,658 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:56:01,658 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:56:01,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:56:01,658 DEBUG [Thread-3183] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:56:01,658 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:56:01,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:56:01,693 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:01,693 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 404 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57202 deadline: 1681606621690, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:01,694 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:01,694 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 392 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57268 deadline: 1681606621691, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:01,758 DEBUG [Thread-3183] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:56:01,758 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:56:01,760 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:56:01,761 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:56:01,761 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:56:01,761 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:56:01,761 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:56:01,762 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:56:01,762 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:56:01,762 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:01,762 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:01,763 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:56:01,763 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:01,763 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:56:01,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:56:01,764 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:01,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:01,764 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:56:01,764 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:01,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:56:01,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:01,765 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:56:01,765 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:56:01,765 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:56:01,765 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:56:01,765 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:56:01,765 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:56:01,765 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:56:01,765 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:56:01,765 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. 2023-04-16 00:56:01,765 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. started... 2023-04-16 00:56:01,766 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. as already flushing 2023-04-16 00:56:01,766 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. 2023-04-16 00:56:01,766 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:56:01,766 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:56:01,766 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:01,766 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:56:01,766 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:56:01,769 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:01,769 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:01,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:01,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:01,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:01,769 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:01,769 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:56:01,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:01,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:01,771 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=897 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp/A/790f2d4a93c046c9b89f374b29f440f5 2023-04-16 00:56:01,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:01,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:01,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:01,774 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:56:01,774 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4177f1a0[Count = 0] remaining members to acquire global barrier 2023-04-16 00:56:01,774 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:56:01,774 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:01,775 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:01,776 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:01,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:01,776 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:56:01,776 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:56:01,776 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:56:01,780 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:01,781 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:56:01,781 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:56:01,781 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:56:01,781 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:56:01,781 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:56:01,781 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:56:01,781 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:56:01,781 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:56:01,781 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:56:01,781 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:56:01,788 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:01,788 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:56:01,788 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:01,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:01,788 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:01,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:01,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:01,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:01,788 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:56:01,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:56:01,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:01,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:01,789 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:01,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:01,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:01,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:01,790 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:01,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:01,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:01,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:01,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:01,793 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:01,793 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:01,793 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:01,793 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:01,793 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:56:01,793 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:01,793 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:01,793 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:01,793 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:56:01,793 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:56:01,794 DEBUG [Thread-3183] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:56:01,794 DEBUG [Thread-3183] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:56:01,794 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:56:01,794 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:56:01,794 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:56:01,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:56:01,794 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:56:01,795 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:56:01,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:56:01,894 DEBUG [Thread-3183] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:56:01,894 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:56:01,896 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:56:01,897 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:56:01,897 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:56:01,897 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:56:01,897 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:56:01,897 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:56:01,897 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:56:01,898 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:01,898 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:01,899 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:56:01,899 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:01,899 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:56:01,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:56:01,899 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:01,899 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:56:01,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:01,900 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:01,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:56:01,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:01,900 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:56:01,900 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:56:01,900 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:56:01,900 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:56:01,900 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:56:01,900 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:56:01,901 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:56:01,901 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:56:01,901 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. 2023-04-16 00:56:01,901 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. started... 2023-04-16 00:56:01,901 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. as already flushing 2023-04-16 00:56:01,901 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. 2023-04-16 00:56:01,901 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:56:01,901 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:56:01,901 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:01,901 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:56:01,901 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:56:01,902 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:01,902 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:01,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:01,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:01,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:01,903 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:01,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:01,903 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:56:01,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:01,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:01,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:01,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:01,906 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:56:01,906 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6dbf2693[Count = 0] remaining members to acquire global barrier 2023-04-16 00:56:01,906 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:56:01,906 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:01,910 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:01,910 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:01,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:01,910 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:56:01,910 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:56:01,910 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:56:01,911 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:56:01,911 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:01,911 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:56:01,912 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:56:01,912 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:56:01,912 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:56:01,912 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:56:01,912 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:56:01,912 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:56:01,912 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:56:01,912 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:56:01,913 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:01,913 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:01,913 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:56:01,913 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:01,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:01,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:01,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:01,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:01,914 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:56:01,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:56:01,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:01,914 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:01,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:01,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:01,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:01,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:01,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:01,915 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:01,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:01,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:01,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:01,919 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:56:01,919 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:01,919 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:56:01,919 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:56:01,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:56:01,919 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:56:01,919 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:56:01,919 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:56:01,919 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:01,919 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:01,919 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:56:01,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:56:01,919 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:01,919 DEBUG [Thread-3183] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:56:01,919 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:01,919 DEBUG [Thread-3183] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:56:01,919 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:01,920 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:01,999 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:01,999 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 406 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57202 deadline: 1681606621996, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:01,999 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:02,000 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 394 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57268 deadline: 1681606621996, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:02,020 DEBUG [Thread-3183] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:56:02,020 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:56:02,021 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:56:02,022 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:56:02,022 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:56:02,022 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:56:02,022 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:56:02,022 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:56:02,023 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:56:02,023 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:02,023 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:02,026 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:56:02,026 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:02,026 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:56:02,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:56:02,026 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:02,026 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:56:02,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:02,027 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:02,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:56:02,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:02,027 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:56:02,027 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:56:02,027 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:56:02,027 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:56:02,027 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:56:02,027 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:56:02,027 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:56:02,028 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:56:02,028 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. 2023-04-16 00:56:02,028 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. started... 2023-04-16 00:56:02,028 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. as already flushing 2023-04-16 00:56:02,028 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. 2023-04-16 00:56:02,028 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:56:02,028 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:56:02,028 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:02,028 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:56:02,028 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:56:02,030 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:02,030 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:02,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:02,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:02,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:02,030 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:02,030 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:56:02,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:02,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:02,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:02,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:02,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:02,033 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:56:02,033 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@176c2cfe[Count = 0] remaining members to acquire global barrier 2023-04-16 00:56:02,033 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:56:02,033 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:02,034 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:02,034 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:02,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:02,034 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:02,034 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:56:02,034 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:56:02,034 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:56:02,034 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:56:02,035 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:02,035 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 214 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57254 deadline: 1681606622031, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:02,035 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:56:02,035 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:56:02,035 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:56:02,037 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:02,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:02,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:02,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:02,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:02,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:02,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:02,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:02,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:02,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:02,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:02,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:56:02,039 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:56:02,039 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:56:02,039 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:56:02,039 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:56:02,039 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:56:02,039 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:56:02,043 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:02,043 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:56:02,043 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:02,043 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:02,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:02,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:02,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:02,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:02,044 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:56:02,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:56:02,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:02,044 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:02,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:02,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:02,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:02,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:02,045 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:02,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:02,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:02,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:02,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:02,048 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:02,048 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:02,049 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:02,049 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:02,049 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:02,049 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:56:02,049 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:56:02,049 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:56:02,049 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:56:02,049 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:56:02,049 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:56:02,049 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:02,049 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:02,049 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:56:02,049 DEBUG [Thread-3183] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:56:02,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:56:02,049 DEBUG [Thread-3183] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:56:02,049 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:56:02,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:56:02,149 DEBUG [Thread-3183] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:56:02,150 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:56:02,151 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:56:02,153 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:56:02,153 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:56:02,153 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:56:02,153 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:56:02,153 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:56:02,153 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:56:02,153 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:02,153 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:02,154 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:56:02,154 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:56:02,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:56:02,154 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:02,155 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:02,155 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:56:02,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:02,155 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:02,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:56:02,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:02,156 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:56:02,156 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:56:02,156 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:56:02,156 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:56:02,156 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:56:02,156 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:56:02,160 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:56:02,160 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:56:02,160 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. 2023-04-16 00:56:02,160 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. started... 2023-04-16 00:56:02,160 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. as already flushing 2023-04-16 00:56:02,160 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. 2023-04-16 00:56:02,160 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:56:02,160 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:56:02,160 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:02,160 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:56:02,160 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:56:02,163 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:02,163 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:02,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:02,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:02,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:02,163 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:02,163 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:56:02,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:02,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:02,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:02,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:02,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:02,165 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:56:02,165 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1503de37[Count = 0] remaining members to acquire global barrier 2023-04-16 00:56:02,165 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:56:02,166 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:02,168 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:02,168 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:02,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:02,168 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:56:02,168 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:56:02,168 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:56:02,168 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:02,168 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:56:02,170 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:02,170 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:56:02,170 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:56:02,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:02,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:02,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:02,170 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:56:02,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:02,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:02,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:02,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:02,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:02,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:02,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:02,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:56:02,176 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:56:02,176 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:56:02,176 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:56:02,176 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:56:02,176 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:56:02,176 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:56:02,177 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:02,177 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:56:02,177 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:02,177 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:02,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:02,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:02,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:02,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:02,178 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:56:02,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:56:02,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:02,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:02,179 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:02,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:02,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:02,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:02,180 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:02,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:02,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:02,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:02,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:02,182 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:02,182 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:56:02,182 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:56:02,182 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:02,182 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:02,182 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:56:02,182 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:56:02,182 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:02,182 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:02,182 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:02,182 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:02,182 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:56:02,183 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:56:02,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:56:02,183 DEBUG [Thread-3183] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:56:02,183 DEBUG [Thread-3183] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:56:02,183 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:56:02,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:56:02,198 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=897 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp/B/1c2fa7f9783c42f1acc17dfee2c1511e 2023-04-16 00:56:02,283 DEBUG [Thread-3183] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:56:02,284 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:56:02,285 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:56:02,286 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:56:02,286 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:56:02,286 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:56:02,286 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:56:02,286 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:56:02,286 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:56:02,287 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:02,287 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:02,289 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:56:02,289 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:02,289 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:56:02,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:56:02,289 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:02,289 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:56:02,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:02,290 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:02,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:56:02,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:02,290 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:56:02,290 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:56:02,290 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:56:02,290 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:56:02,290 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:56:02,290 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:56:02,290 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:56:02,290 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:56:02,290 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. 2023-04-16 00:56:02,291 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. started... 2023-04-16 00:56:02,291 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. as already flushing 2023-04-16 00:56:02,291 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. 2023-04-16 00:56:02,291 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:56:02,291 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:56:02,291 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:02,291 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:56:02,291 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:56:02,292 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:02,292 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:02,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:02,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:02,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:02,293 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:02,293 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:56:02,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:02,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:02,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:02,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:02,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:02,294 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:56:02,294 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6b07d7c6[Count = 0] remaining members to acquire global barrier 2023-04-16 00:56:02,294 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:56:02,295 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:02,295 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:02,295 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:02,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:02,296 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:56:02,296 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:56:02,296 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:56:02,296 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:02,296 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:56:02,300 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:02,300 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:56:02,300 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:56:02,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:02,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:02,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:02,301 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:56:02,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:02,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:02,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:02,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:02,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:02,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:02,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:02,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:56:02,303 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:56:02,303 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:56:02,303 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:56:02,303 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:56:02,303 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:56:02,303 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:56:02,304 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:02,304 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:56:02,304 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:02,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:02,304 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:02,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:02,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:02,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:02,305 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:56:02,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:56:02,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:02,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:02,305 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:02,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:02,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:02,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:02,306 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:02,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:02,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:02,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:02,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:02,308 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:56:02,308 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:56:02,308 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:56:02,308 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:56:02,308 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:56:02,308 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:56:02,309 DEBUG [Thread-3183] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:56:02,309 DEBUG [Thread-3183] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:56:02,310 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:02,310 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:02,310 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:02,310 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:02,310 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:02,310 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:02,310 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:02,310 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:56:02,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:56:02,310 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:56:02,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:56:02,409 DEBUG [Thread-3183] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:56:02,410 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:56:02,411 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:56:02,412 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:56:02,412 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:56:02,412 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:56:02,412 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:56:02,412 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:56:02,412 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:56:02,413 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:02,413 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:02,414 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:02,414 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:56:02,414 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:56:02,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:56:02,415 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:02,415 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:56:02,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:02,415 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:02,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:56:02,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:02,416 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:56:02,416 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:56:02,416 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:56:02,416 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:56:02,416 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:56:02,416 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:56:02,416 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:56:02,416 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:56:02,416 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. 2023-04-16 00:56:02,416 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. started... 2023-04-16 00:56:02,417 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. as already flushing 2023-04-16 00:56:02,417 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. 2023-04-16 00:56:02,417 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:56:02,417 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:56:02,417 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:02,417 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:56:02,417 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:56:02,420 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:02,420 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:02,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:02,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:02,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:02,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:02,421 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:02,421 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:56:02,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:02,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:02,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:02,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:02,423 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:56:02,424 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7f18ff8b[Count = 0] remaining members to acquire global barrier 2023-04-16 00:56:02,424 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:56:02,424 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:02,425 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:02,425 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:02,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:02,425 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:56:02,425 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:56:02,425 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:02,425 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:56:02,425 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:56:02,426 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:56:02,426 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:56:02,426 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:02,426 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:56:02,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:02,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:02,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:02,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:02,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:02,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:02,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:02,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:02,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:02,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:02,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:56:02,429 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:56:02,429 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:56:02,429 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:56:02,429 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:56:02,429 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:56:02,429 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:56:02,431 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:02,431 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:02,431 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:02,431 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:56:02,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:02,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:02,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:02,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:02,431 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:02,431 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:56:02,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:56:02,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:02,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:02,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:02,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:02,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:02,433 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:02,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:02,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:02,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:02,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:02,435 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:56:02,435 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:56:02,436 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:56:02,436 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:02,436 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:02,436 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:02,436 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:02,436 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:02,436 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:02,436 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:02,436 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:56:02,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:56:02,436 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:56:02,436 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:56:02,436 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:56:02,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:56:02,436 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:56:02,436 DEBUG [Thread-3183] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:56:02,437 DEBUG [Thread-3183] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:56:02,510 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:02,510 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 408 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57202 deadline: 1681606622504, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:02,510 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:02,510 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 396 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57268 deadline: 1681606622504, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:02,537 DEBUG [Thread-3183] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:56:02,537 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:56:02,539 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:56:02,540 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:56:02,540 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:56:02,540 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:56:02,540 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:56:02,540 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:56:02,540 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:56:02,541 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:02,541 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:02,542 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:56:02,542 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:56:02,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:56:02,542 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:02,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:02,542 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:02,542 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:56:02,543 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:02,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:56:02,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:02,544 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:56:02,544 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:56:02,544 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:56:02,544 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:56:02,544 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:56:02,544 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:56:02,545 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:56:02,545 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:56:02,545 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. 2023-04-16 00:56:02,545 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. started... 2023-04-16 00:56:02,545 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. as already flushing 2023-04-16 00:56:02,545 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. 2023-04-16 00:56:02,545 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:56:02,545 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:56:02,545 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:02,545 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:56:02,545 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:56:02,546 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:02,546 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:02,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:02,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:02,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:02,547 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:02,547 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:56:02,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:02,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:02,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:02,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:02,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:02,548 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:56:02,548 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@9e5274a[Count = 0] remaining members to acquire global barrier 2023-04-16 00:56:02,548 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:56:02,548 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:02,549 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:02,549 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:02,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:02,549 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:56:02,549 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:56:02,549 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:56:02,550 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:02,550 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:56:02,552 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:02,552 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:56:02,552 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:56:02,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:02,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:02,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:02,552 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:56:02,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:02,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:02,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:02,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:02,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:02,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:02,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:02,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:56:02,554 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:56:02,556 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:56:02,556 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:56:02,556 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:56:02,556 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:56:02,556 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:56:02,557 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:02,557 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:02,557 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:56:02,557 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:02,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:02,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:02,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:02,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:02,558 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:56:02,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:56:02,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:02,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:02,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:02,559 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:02,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:02,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:02,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:02,560 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:02,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:02,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:02,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:02,562 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:56:02,562 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:56:02,562 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:56:02,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:56:02,563 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:56:02,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:56:02,563 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:02,563 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:02,563 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:02,564 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:02,564 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:02,564 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:02,564 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:02,564 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:56:02,564 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:56:02,564 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:56:02,564 DEBUG [Thread-3183] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:56:02,564 DEBUG [Thread-3183] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:56:02,621 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=897 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp/C/9062c7446c1547379c655275d9bcc2d8 2023-04-16 00:56:02,624 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp/A/790f2d4a93c046c9b89f374b29f440f5 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/A/790f2d4a93c046c9b89f374b29f440f5 2023-04-16 00:56:02,627 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/A/790f2d4a93c046c9b89f374b29f440f5, entries=200, sequenceid=897, filesize=14.3 K 2023-04-16 00:56:02,628 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp/B/1c2fa7f9783c42f1acc17dfee2c1511e as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/B/1c2fa7f9783c42f1acc17dfee2c1511e 2023-04-16 00:56:02,631 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/B/1c2fa7f9783c42f1acc17dfee2c1511e, entries=150, sequenceid=897, filesize=11.9 K 2023-04-16 00:56:02,631 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp/C/9062c7446c1547379c655275d9bcc2d8 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/C/9062c7446c1547379c655275d9bcc2d8 2023-04-16 00:56:02,634 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/C/9062c7446c1547379c655275d9bcc2d8, entries=150, sequenceid=897, filesize=11.9 K 2023-04-16 00:56:02,635 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~380.25 KB/389376, currentSize=46.96 KB/48090 for 0abd14932d4ec4e5fdc6177f2c7cc55c in 1304ms, sequenceid=897, compaction requested=false 2023-04-16 00:56:02,635 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 0abd14932d4ec4e5fdc6177f2c7cc55c: 2023-04-16 00:56:02,664 DEBUG [Thread-3183] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:56:02,665 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:56:02,666 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:56:02,667 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:56:02,667 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:56:02,667 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:56:02,668 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:56:02,668 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:56:02,668 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:56:02,668 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:02,668 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:02,669 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:56:02,669 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:02,669 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:56:02,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:56:02,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:02,670 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:02,670 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:56:02,670 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:02,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:56:02,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:02,671 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:56:02,671 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:56:02,671 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:56:02,671 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:56:02,671 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:56:02,671 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:56:02,671 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:56:02,671 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:56:02,671 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. 2023-04-16 00:56:02,671 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. started... 2023-04-16 00:56:02,671 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 0abd14932d4ec4e5fdc6177f2c7cc55c 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-16 00:56:02,672 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0abd14932d4ec4e5fdc6177f2c7cc55c, store=A 2023-04-16 00:56:02,672 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:56:02,672 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0abd14932d4ec4e5fdc6177f2c7cc55c, store=B 2023-04-16 00:56:02,672 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:56:02,672 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0abd14932d4ec4e5fdc6177f2c7cc55c, store=C 2023-04-16 00:56:02,672 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:56:02,686 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=910 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp/A/5322b49c1d9e4da597eb293b436af71f 2023-04-16 00:56:02,700 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=910 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp/B/eadeb1fca1a64eec954fc915834d8303 2023-04-16 00:56:02,912 DEBUG [Thread-3190] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x211582f2 to 127.0.0.1:53150 2023-04-16 00:56:02,912 DEBUG [Thread-3190] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-16 00:56:02,914 DEBUG [Thread-3184] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x67346c47 to 127.0.0.1:53150 2023-04-16 00:56:02,914 DEBUG [Thread-3184] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-16 00:56:02,915 DEBUG [Thread-3188] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5f5363d8 to 127.0.0.1:53150 2023-04-16 00:56:02,915 DEBUG [Thread-3188] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-16 00:56:02,917 DEBUG [Thread-3186] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x426cb82a to 127.0.0.1:53150 2023-04-16 00:56:02,917 DEBUG [Thread-3186] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-16 00:56:02,917 DEBUG [Thread-3192] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x65816b6e to 127.0.0.1:53150 2023-04-16 00:56:02,917 DEBUG [Thread-3192] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-16 00:56:03,042 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(9158): Flush requested on 0abd14932d4ec4e5fdc6177f2c7cc55c 2023-04-16 00:56:03,042 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. as already flushing 2023-04-16 00:56:03,042 DEBUG [Thread-3175] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x27f2d997 to 127.0.0.1:53150 2023-04-16 00:56:03,042 DEBUG [Thread-3175] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-16 00:56:03,111 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=910 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp/C/0a85397b09a04b3ca704b9c7de410d79 2023-04-16 00:56:03,115 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp/A/5322b49c1d9e4da597eb293b436af71f as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/A/5322b49c1d9e4da597eb293b436af71f 2023-04-16 00:56:03,119 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/A/5322b49c1d9e4da597eb293b436af71f, entries=150, sequenceid=910, filesize=11.9 K 2023-04-16 00:56:03,120 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp/B/eadeb1fca1a64eec954fc915834d8303 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/B/eadeb1fca1a64eec954fc915834d8303 2023-04-16 00:56:03,123 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/B/eadeb1fca1a64eec954fc915834d8303, entries=150, sequenceid=910, filesize=11.9 K 2023-04-16 00:56:03,123 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp/C/0a85397b09a04b3ca704b9c7de410d79 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/C/0a85397b09a04b3ca704b9c7de410d79 2023-04-16 00:56:03,126 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/C/0a85397b09a04b3ca704b9c7de410d79, entries=150, sequenceid=910, filesize=11.9 K 2023-04-16 00:56:03,127 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=6.71 KB/6870 for 0abd14932d4ec4e5fdc6177f2c7cc55c in 456ms, sequenceid=910, compaction requested=true 2023-04-16 00:56:03,127 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 0abd14932d4ec4e5fdc6177f2c7cc55c: 2023-04-16 00:56:03,127 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. 2023-04-16 00:56:03,127 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:56:03,127 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:56:03,127 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:03,127 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:56:03,127 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:56:03,129 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:03,129 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:03,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:03,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:03,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:03,129 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:03,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:03,129 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:56:03,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:03,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:03,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:03,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:03,130 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:56:03,130 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@21ca227[Count = 0] remaining members to acquire global barrier 2023-04-16 00:56:03,130 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:56:03,130 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:03,131 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:03,131 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:03,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:03,131 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:56:03,131 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:56:03,131 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:56:03,131 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:03,131 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:56:03,137 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:03,137 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:56:03,137 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:56:03,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:03,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:03,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:03,137 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:56:03,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:03,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:03,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:03,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:03,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:03,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:03,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:03,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:56:03,139 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:56:03,139 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:56:03,139 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:56:03,139 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:56:03,139 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:56:03,139 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:56:03,141 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:03,141 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:03,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:03,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:03,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:03,141 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:03,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:03,141 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:56:03,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:03,141 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:56:03,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:56:03,141 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:03,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:03,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:03,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:03,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:03,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:03,142 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:03,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:03,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:03,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:03,144 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:03,144 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:56:03,144 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:03,144 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:56:03,144 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:56:03,144 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:56:03,144 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:03,144 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:56:03,144 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:03,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:56:03,145 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:03,144 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:56:03,145 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:03,145 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:03,145 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:56:03,145 DEBUG [Thread-3183] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:56:03,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:56:03,145 DEBUG [Thread-3183] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:56:03,245 DEBUG [Thread-3183] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:56:03,246 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:56:03,513 DEBUG [Thread-3173] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x501c5a88 to 127.0.0.1:53150 2023-04-16 00:56:03,514 DEBUG [Thread-3173] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-16 00:56:03,521 DEBUG [Thread-3179] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1c29757d to 127.0.0.1:53150 2023-04-16 00:56:03,521 DEBUG [Thread-3179] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-16 00:56:05,538 DEBUG [Thread-3177] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x34ad4aa8 to 127.0.0.1:53150 2023-04-16 00:56:05,538 DEBUG [Thread-3177] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-16 00:56:09,206 DEBUG [Thread-3181] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x06edd017 to 127.0.0.1:53150 2023-04-16 00:56:09,206 DEBUG [Thread-3181] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-16 00:56:09,206 INFO [Listener at localhost/33477] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-16 00:56:09,206 INFO [Listener at localhost/33477] hbase.AcidGuaranteesTestTool(394): wrote 261 2023-04-16 00:56:09,206 INFO [Listener at localhost/33477] hbase.AcidGuaranteesTestTool(394): wrote 117 2023-04-16 00:56:09,207 INFO [Listener at localhost/33477] hbase.AcidGuaranteesTestTool(394): wrote 28 2023-04-16 00:56:09,207 INFO [Listener at localhost/33477] hbase.AcidGuaranteesTestTool(394): wrote 249 2023-04-16 00:56:09,207 INFO [Listener at localhost/33477] hbase.AcidGuaranteesTestTool(394): wrote 109 2023-04-16 00:56:09,207 INFO [Listener at localhost/33477] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-16 00:56:09,207 INFO [Listener at localhost/33477] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-16 00:56:09,207 INFO [Listener at localhost/33477] hbase.AcidGuaranteesTestTool(402): scanned 1787 2023-04-16 00:56:09,207 INFO [Listener at localhost/33477] hbase.AcidGuaranteesTestTool(403): verified 5361 rows 2023-04-16 00:56:09,207 INFO [Listener at localhost/33477] hbase.AcidGuaranteesTestTool(402): scanned 1793 2023-04-16 00:56:09,207 INFO [Listener at localhost/33477] hbase.AcidGuaranteesTestTool(403): verified 5379 rows 2023-04-16 00:56:09,207 INFO [Listener at localhost/33477] hbase.AcidGuaranteesTestTool(402): scanned 1814 2023-04-16 00:56:09,207 INFO [Listener at localhost/33477] hbase.AcidGuaranteesTestTool(403): verified 5442 rows 2023-04-16 00:56:09,207 INFO [Listener at localhost/33477] hbase.AcidGuaranteesTestTool(402): scanned 1792 2023-04-16 00:56:09,207 INFO [Listener at localhost/33477] hbase.AcidGuaranteesTestTool(403): verified 5376 rows 2023-04-16 00:56:09,207 INFO [Listener at localhost/33477] hbase.AcidGuaranteesTestTool(402): scanned 1811 2023-04-16 00:56:09,207 INFO [Listener at localhost/33477] hbase.AcidGuaranteesTestTool(403): verified 5430 rows 2023-04-16 00:56:09,207 INFO [Listener at localhost/33477] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-16 00:56:09,207 DEBUG [Listener at localhost/33477] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5e372785 to 127.0.0.1:53150 2023-04-16 00:56:09,207 DEBUG [Listener at localhost/33477] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-16 00:56:09,209 INFO [Listener at localhost/33477] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-16 00:56:09,209 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable TestAcidGuarantees 2023-04-16 00:56:09,210 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure2.ProcedureExecutor(1029): Stored pid=50, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-16 00:56:09,213 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-16 00:56:09,213 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681606569213"}]},"ts":"1681606569213"} 2023-04-16 00:56:09,214 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-16 00:56:09,216 INFO [PEWorker-4] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-16 00:56:09,217 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=51, ppid=50, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=0abd14932d4ec4e5fdc6177f2c7cc55c, UNASSIGN}] 2023-04-16 00:56:09,218 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=51, ppid=50, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=0abd14932d4ec4e5fdc6177f2c7cc55c, UNASSIGN 2023-04-16 00:56:09,219 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=51 updating hbase:meta row=0abd14932d4ec4e5fdc6177f2c7cc55c, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:09,219 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681606569219"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681606569219"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681606569219"}]},"ts":"1681606569219"} 2023-04-16 00:56:09,219 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=52, ppid=51, state=RUNNABLE; CloseRegionProcedure 0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217}] 2023-04-16 00:56:09,314 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-16 00:56:09,371 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 0abd14932d4ec4e5fdc6177f2c7cc55c 2023-04-16 00:56:09,371 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 0abd14932d4ec4e5fdc6177f2c7cc55c, disabling compactions & flushes 2023-04-16 00:56:09,371 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. 2023-04-16 00:56:09,371 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. 2023-04-16 00:56:09,371 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. after waiting 0 ms 2023-04-16 00:56:09,371 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. 2023-04-16 00:56:09,371 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 0abd14932d4ec4e5fdc6177f2c7cc55c 3/3 column families, dataSize=33.54 KB heapSize=100.36 KB 2023-04-16 00:56:09,372 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0abd14932d4ec4e5fdc6177f2c7cc55c, store=A 2023-04-16 00:56:09,372 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:56:09,372 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0abd14932d4ec4e5fdc6177f2c7cc55c, store=B 2023-04-16 00:56:09,372 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:56:09,372 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0abd14932d4ec4e5fdc6177f2c7cc55c, store=C 2023-04-16 00:56:09,372 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:56:09,514 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-16 00:56:09,780 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=918 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp/A/8e9c0baefe0d4dfb8990941574b8da20 2023-04-16 00:56:09,815 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-16 00:56:10,189 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=918 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp/B/e62c19adf57944aeae231befa481ee3e 2023-04-16 00:56:10,316 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-16 00:56:10,597 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=918 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp/C/abe08fc104fd41c0a091f0b70e27dce8 2023-04-16 00:56:10,600 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp/A/8e9c0baefe0d4dfb8990941574b8da20 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/A/8e9c0baefe0d4dfb8990941574b8da20 2023-04-16 00:56:10,602 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/A/8e9c0baefe0d4dfb8990941574b8da20, entries=100, sequenceid=918, filesize=9.5 K 2023-04-16 00:56:10,603 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp/B/e62c19adf57944aeae231befa481ee3e as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/B/e62c19adf57944aeae231befa481ee3e 2023-04-16 00:56:10,606 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/B/e62c19adf57944aeae231befa481ee3e, entries=100, sequenceid=918, filesize=9.5 K 2023-04-16 00:56:10,606 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/.tmp/C/abe08fc104fd41c0a091f0b70e27dce8 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/C/abe08fc104fd41c0a091f0b70e27dce8 2023-04-16 00:56:10,609 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/C/abe08fc104fd41c0a091f0b70e27dce8, entries=100, sequenceid=918, filesize=9.5 K 2023-04-16 00:56:10,610 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~100.31 KB/102720, currentSize=0 B/0 for 0abd14932d4ec4e5fdc6177f2c7cc55c in 1239ms, sequenceid=918, compaction requested=true 2023-04-16 00:56:10,634 DEBUG [StoreCloser-TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/A/71418beef49441f398ffb9fdee497861, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/A/f7a21da1edc6453a87700f88d4768b2c, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/A/a536085eda52409db66262bfc6d27349, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/A/e837534d24084370aaa4880531ac951f, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/A/cb45045553bf4063b4c5318114b406e7, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/A/a7aea499e2c74e8ea2c5d13b76636911, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/A/cdc912a5a9c948e7b803f03edf8cfa6e, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/A/8481f81a8bbc45fda145aeed58997fd7, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/A/7296aeea67a740a3baa8203cdefc1ba8, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/A/be0f2522251c4690a9a527d58be2d4ec, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/A/6f4c902e8e3b4557999b24c5aa5061e4, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/A/bfea04ef6d5c40ab826f04ad1a891b09, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/A/6d59006731764706bee1a03812236949, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/A/cba347c854c54e6091c926834313b1c9, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/A/e8351b101df84b41b3666061b9a9d395, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/A/b627f7424e304af0a35e600740b3395d, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/A/99f7e63612134d64b8ccc795b3969ad8, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/A/9aca28d6a62f4703a365acdbc5622b5e, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/A/af6f6d3322aa401a941aaed00e280519, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/A/4653c4bed25c4a069797b7ac706e310b, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/A/d4bad50742b240f58175e98280b393dd, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/A/2a22bf6024084919b07d5dc7eafd9f8c, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/A/fab05b92632a4535bb18e46a57bc4e65, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/A/e3aa8619e99e4cbf9db6cefc4aab466a, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/A/adaed1c21e2047d785ef510f855c706f, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/A/c66ee34508b5418f8940b8c62c52e14f, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/A/cfc31dc4c0f04e1b8bb19ad5917e9b08, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/A/1b751a155a694b6abf2cb0d6440c07df, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/A/d416eff326974c3bacb78ccaae62834d, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/A/d249d3f6293540518b0382a8c2e1f37e, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/A/87af756bff9b4afdbcf96348945ae7c4, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/A/80d19764c04b42689408d9015a17b1e7, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/A/7b0c3d5dda444252b5f5ec4b1a0017f9, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/A/19242e54d92c46e8a518c507507c154f, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/A/a72707feed22489a88bd032ea60dcfee, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/A/e493d1145a3e4ffdb15b77bc9b9e00ec, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/A/dd7214cf35164c9e97fe1d6b6b2cf90c, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/A/40fc84d7e7b9466a866f046bdf99118c, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/A/c1e14b4655e441bfacb8dd781b679359, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/A/9ed538e97e95423f8ea10c10c7d87c63, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/A/2e2bba7d0b8c44e59a06bfe2fb1ad054, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/A/4db97e3ae479483a92bc130f4f04da2e, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/A/29812f77ac024216be8f2520f5a9393e, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/A/5dae8985a1134867bb5a9dcf3e7603dd, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/A/6add4c0913b048998ff99cef9bfbc87f, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/A/d9acb3e4c4994d8c976105a6d11ae91b] to archive 2023-04-16 00:56:10,635 DEBUG [StoreCloser-TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-16 00:56:10,636 DEBUG [StoreCloser-TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/A/71418beef49441f398ffb9fdee497861 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/A/71418beef49441f398ffb9fdee497861 2023-04-16 00:56:10,638 DEBUG [StoreCloser-TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/A/f7a21da1edc6453a87700f88d4768b2c to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/A/f7a21da1edc6453a87700f88d4768b2c 2023-04-16 00:56:10,639 DEBUG [StoreCloser-TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/A/a536085eda52409db66262bfc6d27349 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/A/a536085eda52409db66262bfc6d27349 2023-04-16 00:56:10,640 DEBUG [StoreCloser-TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/A/e837534d24084370aaa4880531ac951f to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/A/e837534d24084370aaa4880531ac951f 2023-04-16 00:56:10,641 DEBUG [StoreCloser-TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/A/cb45045553bf4063b4c5318114b406e7 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/A/cb45045553bf4063b4c5318114b406e7 2023-04-16 00:56:10,642 DEBUG [StoreCloser-TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/A/a7aea499e2c74e8ea2c5d13b76636911 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/A/a7aea499e2c74e8ea2c5d13b76636911 2023-04-16 00:56:10,642 DEBUG [StoreCloser-TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/A/cdc912a5a9c948e7b803f03edf8cfa6e to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/A/cdc912a5a9c948e7b803f03edf8cfa6e 2023-04-16 00:56:10,643 DEBUG [StoreCloser-TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/A/8481f81a8bbc45fda145aeed58997fd7 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/A/8481f81a8bbc45fda145aeed58997fd7 2023-04-16 00:56:10,644 DEBUG [StoreCloser-TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/A/7296aeea67a740a3baa8203cdefc1ba8 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/A/7296aeea67a740a3baa8203cdefc1ba8 2023-04-16 00:56:10,645 DEBUG [StoreCloser-TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/A/be0f2522251c4690a9a527d58be2d4ec to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/A/be0f2522251c4690a9a527d58be2d4ec 2023-04-16 00:56:10,646 DEBUG [StoreCloser-TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/A/6f4c902e8e3b4557999b24c5aa5061e4 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/A/6f4c902e8e3b4557999b24c5aa5061e4 2023-04-16 00:56:10,647 DEBUG [StoreCloser-TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/A/bfea04ef6d5c40ab826f04ad1a891b09 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/A/bfea04ef6d5c40ab826f04ad1a891b09 2023-04-16 00:56:10,648 DEBUG [StoreCloser-TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/A/6d59006731764706bee1a03812236949 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/A/6d59006731764706bee1a03812236949 2023-04-16 00:56:10,649 DEBUG [StoreCloser-TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/A/cba347c854c54e6091c926834313b1c9 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/A/cba347c854c54e6091c926834313b1c9 2023-04-16 00:56:10,650 DEBUG [StoreCloser-TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/A/e8351b101df84b41b3666061b9a9d395 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/A/e8351b101df84b41b3666061b9a9d395 2023-04-16 00:56:10,651 DEBUG [StoreCloser-TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/A/b627f7424e304af0a35e600740b3395d to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/A/b627f7424e304af0a35e600740b3395d 2023-04-16 00:56:10,652 DEBUG [StoreCloser-TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/A/99f7e63612134d64b8ccc795b3969ad8 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/A/99f7e63612134d64b8ccc795b3969ad8 2023-04-16 00:56:10,653 DEBUG [StoreCloser-TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/A/9aca28d6a62f4703a365acdbc5622b5e to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/A/9aca28d6a62f4703a365acdbc5622b5e 2023-04-16 00:56:10,653 DEBUG [StoreCloser-TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/A/af6f6d3322aa401a941aaed00e280519 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/A/af6f6d3322aa401a941aaed00e280519 2023-04-16 00:56:10,654 DEBUG [StoreCloser-TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/A/4653c4bed25c4a069797b7ac706e310b to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/A/4653c4bed25c4a069797b7ac706e310b 2023-04-16 00:56:10,655 DEBUG [StoreCloser-TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/A/d4bad50742b240f58175e98280b393dd to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/A/d4bad50742b240f58175e98280b393dd 2023-04-16 00:56:10,656 DEBUG [StoreCloser-TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/A/2a22bf6024084919b07d5dc7eafd9f8c to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/A/2a22bf6024084919b07d5dc7eafd9f8c 2023-04-16 00:56:10,657 DEBUG [StoreCloser-TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/A/fab05b92632a4535bb18e46a57bc4e65 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/A/fab05b92632a4535bb18e46a57bc4e65 2023-04-16 00:56:10,658 DEBUG [StoreCloser-TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/A/e3aa8619e99e4cbf9db6cefc4aab466a to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/A/e3aa8619e99e4cbf9db6cefc4aab466a 2023-04-16 00:56:10,659 DEBUG [StoreCloser-TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/A/adaed1c21e2047d785ef510f855c706f to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/A/adaed1c21e2047d785ef510f855c706f 2023-04-16 00:56:10,660 DEBUG [StoreCloser-TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/A/c66ee34508b5418f8940b8c62c52e14f to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/A/c66ee34508b5418f8940b8c62c52e14f 2023-04-16 00:56:10,661 DEBUG [StoreCloser-TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/A/cfc31dc4c0f04e1b8bb19ad5917e9b08 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/A/cfc31dc4c0f04e1b8bb19ad5917e9b08 2023-04-16 00:56:10,662 DEBUG [StoreCloser-TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/A/1b751a155a694b6abf2cb0d6440c07df to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/A/1b751a155a694b6abf2cb0d6440c07df 2023-04-16 00:56:10,663 DEBUG [StoreCloser-TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/A/d416eff326974c3bacb78ccaae62834d to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/A/d416eff326974c3bacb78ccaae62834d 2023-04-16 00:56:10,664 DEBUG [StoreCloser-TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/A/d249d3f6293540518b0382a8c2e1f37e to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/A/d249d3f6293540518b0382a8c2e1f37e 2023-04-16 00:56:10,665 DEBUG [StoreCloser-TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/A/87af756bff9b4afdbcf96348945ae7c4 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/A/87af756bff9b4afdbcf96348945ae7c4 2023-04-16 00:56:10,667 DEBUG [StoreCloser-TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/A/80d19764c04b42689408d9015a17b1e7 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/A/80d19764c04b42689408d9015a17b1e7 2023-04-16 00:56:10,667 DEBUG [StoreCloser-TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/A/7b0c3d5dda444252b5f5ec4b1a0017f9 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/A/7b0c3d5dda444252b5f5ec4b1a0017f9 2023-04-16 00:56:10,668 DEBUG [StoreCloser-TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/A/19242e54d92c46e8a518c507507c154f to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/A/19242e54d92c46e8a518c507507c154f 2023-04-16 00:56:10,669 DEBUG [StoreCloser-TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/A/a72707feed22489a88bd032ea60dcfee to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/A/a72707feed22489a88bd032ea60dcfee 2023-04-16 00:56:10,670 DEBUG [StoreCloser-TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/A/e493d1145a3e4ffdb15b77bc9b9e00ec to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/A/e493d1145a3e4ffdb15b77bc9b9e00ec 2023-04-16 00:56:10,671 DEBUG [StoreCloser-TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/A/dd7214cf35164c9e97fe1d6b6b2cf90c to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/A/dd7214cf35164c9e97fe1d6b6b2cf90c 2023-04-16 00:56:10,671 DEBUG [StoreCloser-TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/A/40fc84d7e7b9466a866f046bdf99118c to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/A/40fc84d7e7b9466a866f046bdf99118c 2023-04-16 00:56:10,672 DEBUG [StoreCloser-TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/A/c1e14b4655e441bfacb8dd781b679359 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/A/c1e14b4655e441bfacb8dd781b679359 2023-04-16 00:56:10,673 DEBUG [StoreCloser-TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/A/9ed538e97e95423f8ea10c10c7d87c63 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/A/9ed538e97e95423f8ea10c10c7d87c63 2023-04-16 00:56:10,674 DEBUG [StoreCloser-TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/A/2e2bba7d0b8c44e59a06bfe2fb1ad054 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/A/2e2bba7d0b8c44e59a06bfe2fb1ad054 2023-04-16 00:56:10,674 DEBUG [StoreCloser-TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/A/4db97e3ae479483a92bc130f4f04da2e to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/A/4db97e3ae479483a92bc130f4f04da2e 2023-04-16 00:56:10,675 DEBUG [StoreCloser-TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/A/29812f77ac024216be8f2520f5a9393e to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/A/29812f77ac024216be8f2520f5a9393e 2023-04-16 00:56:10,676 DEBUG [StoreCloser-TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/A/5dae8985a1134867bb5a9dcf3e7603dd to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/A/5dae8985a1134867bb5a9dcf3e7603dd 2023-04-16 00:56:10,677 DEBUG [StoreCloser-TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/A/6add4c0913b048998ff99cef9bfbc87f to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/A/6add4c0913b048998ff99cef9bfbc87f 2023-04-16 00:56:10,678 DEBUG [StoreCloser-TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/A/d9acb3e4c4994d8c976105a6d11ae91b to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/A/d9acb3e4c4994d8c976105a6d11ae91b 2023-04-16 00:56:10,681 DEBUG [StoreCloser-TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/B/f4aad6d467e94184902a4a8943a66c2d, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/B/2446ae0deb3c4113b943f0f34e7e4b9a, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/B/87dac584dc4248e28d98db785d1b2d4e, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/B/f0973d70203e443aa76f898d60d85c6b, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/B/784aaa8b6afc4fe58de0a2ebc5d31b38, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/B/134f3fae05af47148ac0f7d284854e20, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/B/4be52027907649ae8dc19e46d5332468, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/B/51638d2847714f59a9001c13cc9b1ddf, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/B/7f05a6d097444efdab1b362d137e0f01, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/B/8c2f5ab27b904c93864bb44be619d59e, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/B/c40a435920d34083a141523f490ea0a6, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/B/0aaf263ff33f44608cd1ff27a4f1c8de, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/B/544c2a6ab4d04325accff040f2e4827d, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/B/4cb6a22b1d824413a2892ccf35d20cd0, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/B/4205e1d340c241bfb527f56828df10e3, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/B/ea3738a001ff4d7494adfc7f72a3b985, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/B/980a4fbe8fc041c4816337e40a02c1b1, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/B/85223b01a4c54d7ca4a5d37ef2deb5ad, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/B/df6dc2d800084e53bebca094f7dfe523, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/B/bf5fcc1a45b44735aff89fb4837aec89, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/B/928f079b7c704dad9afde3a70a9c43f8, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/B/a94f89cd089245449345e46b42f9f598, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/B/26216e227f1a44d2ab0d02b14805c8bd, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/B/fb1aef86d13f456f80880e90bbd71fdb, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/B/24d8518a58b24f75bcbf11fa964162f9, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/B/9dfaeedb80404bbe934f59ebf0eb6615, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/B/1777f7490a1d429e9e462cc26a768f9f, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/B/2a1d52947afa47a18bff012aa82be4dd, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/B/72e46b98fcd348ee9b95584e587f3579, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/B/84f4b2a4eb77486882475d66b97e2de0, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/B/97d133c09cca4f00b5c7feb64ab1871f, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/B/977ff4fdd59146deab38df0aabae3663, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/B/1ab9f6ad31904a5cb6512bbd61005a1d, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/B/96b56dda3b174aaf80e9c8fd418cf46a, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/B/eac3fce8f98b4eaabf995b5dc9ecacb9, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/B/a40971af59fd4d3288b0337f3eea29f4, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/B/dbb5eea9c8dc47a79a432c684793be13, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/B/95dab1902c684fe287c95c641847484f, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/B/c3058cdcd16644c99ff7ba70db719068, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/B/4ba88f818e74441a82b2e3e1206d7295, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/B/bc18294df4f64767888c14fc4206eee1, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/B/dee878d091d14cdb9e799fe52ed2d2f3, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/B/86a8ab0f049e407c95f96d85c35b806f, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/B/2f93bdd985ba43ee9fa43d2d082d8481, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/B/36f805bec13542a78757d775f78a4d24, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/B/896454a8e74247afa37156e1830ad25e] to archive 2023-04-16 00:56:10,682 DEBUG [StoreCloser-TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-16 00:56:10,683 DEBUG [StoreCloser-TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/B/f4aad6d467e94184902a4a8943a66c2d to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/B/f4aad6d467e94184902a4a8943a66c2d 2023-04-16 00:56:10,684 DEBUG [StoreCloser-TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/B/2446ae0deb3c4113b943f0f34e7e4b9a to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/B/2446ae0deb3c4113b943f0f34e7e4b9a 2023-04-16 00:56:10,685 DEBUG [StoreCloser-TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/B/87dac584dc4248e28d98db785d1b2d4e to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/B/87dac584dc4248e28d98db785d1b2d4e 2023-04-16 00:56:10,686 DEBUG [StoreCloser-TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/B/f0973d70203e443aa76f898d60d85c6b to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/B/f0973d70203e443aa76f898d60d85c6b 2023-04-16 00:56:10,687 DEBUG [StoreCloser-TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/B/784aaa8b6afc4fe58de0a2ebc5d31b38 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/B/784aaa8b6afc4fe58de0a2ebc5d31b38 2023-04-16 00:56:10,688 DEBUG [StoreCloser-TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/B/134f3fae05af47148ac0f7d284854e20 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/B/134f3fae05af47148ac0f7d284854e20 2023-04-16 00:56:10,689 DEBUG [StoreCloser-TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/B/4be52027907649ae8dc19e46d5332468 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/B/4be52027907649ae8dc19e46d5332468 2023-04-16 00:56:10,689 DEBUG [StoreCloser-TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/B/51638d2847714f59a9001c13cc9b1ddf to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/B/51638d2847714f59a9001c13cc9b1ddf 2023-04-16 00:56:10,690 DEBUG [StoreCloser-TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/B/7f05a6d097444efdab1b362d137e0f01 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/B/7f05a6d097444efdab1b362d137e0f01 2023-04-16 00:56:10,691 DEBUG [StoreCloser-TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/B/8c2f5ab27b904c93864bb44be619d59e to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/B/8c2f5ab27b904c93864bb44be619d59e 2023-04-16 00:56:10,692 DEBUG [StoreCloser-TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/B/c40a435920d34083a141523f490ea0a6 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/B/c40a435920d34083a141523f490ea0a6 2023-04-16 00:56:10,693 DEBUG [StoreCloser-TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/B/0aaf263ff33f44608cd1ff27a4f1c8de to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/B/0aaf263ff33f44608cd1ff27a4f1c8de 2023-04-16 00:56:10,694 DEBUG [StoreCloser-TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/B/544c2a6ab4d04325accff040f2e4827d to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/B/544c2a6ab4d04325accff040f2e4827d 2023-04-16 00:56:10,695 DEBUG [StoreCloser-TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/B/4cb6a22b1d824413a2892ccf35d20cd0 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/B/4cb6a22b1d824413a2892ccf35d20cd0 2023-04-16 00:56:10,696 DEBUG [StoreCloser-TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/B/4205e1d340c241bfb527f56828df10e3 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/B/4205e1d340c241bfb527f56828df10e3 2023-04-16 00:56:10,696 DEBUG [StoreCloser-TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/B/ea3738a001ff4d7494adfc7f72a3b985 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/B/ea3738a001ff4d7494adfc7f72a3b985 2023-04-16 00:56:10,697 DEBUG [StoreCloser-TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/B/980a4fbe8fc041c4816337e40a02c1b1 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/B/980a4fbe8fc041c4816337e40a02c1b1 2023-04-16 00:56:10,698 DEBUG [StoreCloser-TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/B/85223b01a4c54d7ca4a5d37ef2deb5ad to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/B/85223b01a4c54d7ca4a5d37ef2deb5ad 2023-04-16 00:56:10,699 DEBUG [StoreCloser-TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/B/df6dc2d800084e53bebca094f7dfe523 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/B/df6dc2d800084e53bebca094f7dfe523 2023-04-16 00:56:10,700 DEBUG [StoreCloser-TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/B/bf5fcc1a45b44735aff89fb4837aec89 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/B/bf5fcc1a45b44735aff89fb4837aec89 2023-04-16 00:56:10,701 DEBUG [StoreCloser-TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/B/928f079b7c704dad9afde3a70a9c43f8 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/B/928f079b7c704dad9afde3a70a9c43f8 2023-04-16 00:56:10,701 DEBUG [StoreCloser-TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/B/a94f89cd089245449345e46b42f9f598 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/B/a94f89cd089245449345e46b42f9f598 2023-04-16 00:56:10,702 DEBUG [StoreCloser-TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/B/26216e227f1a44d2ab0d02b14805c8bd to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/B/26216e227f1a44d2ab0d02b14805c8bd 2023-04-16 00:56:10,703 DEBUG [StoreCloser-TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/B/fb1aef86d13f456f80880e90bbd71fdb to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/B/fb1aef86d13f456f80880e90bbd71fdb 2023-04-16 00:56:10,704 DEBUG [StoreCloser-TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/B/24d8518a58b24f75bcbf11fa964162f9 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/B/24d8518a58b24f75bcbf11fa964162f9 2023-04-16 00:56:10,705 DEBUG [StoreCloser-TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/B/9dfaeedb80404bbe934f59ebf0eb6615 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/B/9dfaeedb80404bbe934f59ebf0eb6615 2023-04-16 00:56:10,706 DEBUG [StoreCloser-TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/B/1777f7490a1d429e9e462cc26a768f9f to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/B/1777f7490a1d429e9e462cc26a768f9f 2023-04-16 00:56:10,707 DEBUG [StoreCloser-TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/B/2a1d52947afa47a18bff012aa82be4dd to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/B/2a1d52947afa47a18bff012aa82be4dd 2023-04-16 00:56:10,707 DEBUG [StoreCloser-TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/B/72e46b98fcd348ee9b95584e587f3579 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/B/72e46b98fcd348ee9b95584e587f3579 2023-04-16 00:56:10,708 DEBUG [StoreCloser-TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/B/84f4b2a4eb77486882475d66b97e2de0 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/B/84f4b2a4eb77486882475d66b97e2de0 2023-04-16 00:56:10,709 DEBUG [StoreCloser-TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/B/97d133c09cca4f00b5c7feb64ab1871f to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/B/97d133c09cca4f00b5c7feb64ab1871f 2023-04-16 00:56:10,710 DEBUG [StoreCloser-TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/B/977ff4fdd59146deab38df0aabae3663 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/B/977ff4fdd59146deab38df0aabae3663 2023-04-16 00:56:10,711 DEBUG [StoreCloser-TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/B/1ab9f6ad31904a5cb6512bbd61005a1d to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/B/1ab9f6ad31904a5cb6512bbd61005a1d 2023-04-16 00:56:10,712 DEBUG [StoreCloser-TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/B/96b56dda3b174aaf80e9c8fd418cf46a to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/B/96b56dda3b174aaf80e9c8fd418cf46a 2023-04-16 00:56:10,712 DEBUG [StoreCloser-TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/B/eac3fce8f98b4eaabf995b5dc9ecacb9 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/B/eac3fce8f98b4eaabf995b5dc9ecacb9 2023-04-16 00:56:10,713 DEBUG [StoreCloser-TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/B/a40971af59fd4d3288b0337f3eea29f4 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/B/a40971af59fd4d3288b0337f3eea29f4 2023-04-16 00:56:10,714 DEBUG [StoreCloser-TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/B/dbb5eea9c8dc47a79a432c684793be13 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/B/dbb5eea9c8dc47a79a432c684793be13 2023-04-16 00:56:10,715 DEBUG [StoreCloser-TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/B/95dab1902c684fe287c95c641847484f to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/B/95dab1902c684fe287c95c641847484f 2023-04-16 00:56:10,716 DEBUG [StoreCloser-TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/B/c3058cdcd16644c99ff7ba70db719068 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/B/c3058cdcd16644c99ff7ba70db719068 2023-04-16 00:56:10,717 DEBUG [StoreCloser-TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/B/4ba88f818e74441a82b2e3e1206d7295 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/B/4ba88f818e74441a82b2e3e1206d7295 2023-04-16 00:56:10,718 DEBUG [StoreCloser-TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/B/bc18294df4f64767888c14fc4206eee1 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/B/bc18294df4f64767888c14fc4206eee1 2023-04-16 00:56:10,719 DEBUG [StoreCloser-TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/B/dee878d091d14cdb9e799fe52ed2d2f3 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/B/dee878d091d14cdb9e799fe52ed2d2f3 2023-04-16 00:56:10,719 DEBUG [StoreCloser-TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/B/86a8ab0f049e407c95f96d85c35b806f to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/B/86a8ab0f049e407c95f96d85c35b806f 2023-04-16 00:56:10,721 DEBUG [StoreCloser-TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/B/2f93bdd985ba43ee9fa43d2d082d8481 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/B/2f93bdd985ba43ee9fa43d2d082d8481 2023-04-16 00:56:10,721 DEBUG [StoreCloser-TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/B/36f805bec13542a78757d775f78a4d24 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/B/36f805bec13542a78757d775f78a4d24 2023-04-16 00:56:10,722 DEBUG [StoreCloser-TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/B/896454a8e74247afa37156e1830ad25e to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/B/896454a8e74247afa37156e1830ad25e 2023-04-16 00:56:10,726 DEBUG [StoreCloser-TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/C/a73b04d521c54866abefb4e0094431e3, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/C/330a03ff6a924906b546600756cc25ea, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/C/eae239871d0845019a296960b5d4fd40, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/C/a3847ebac68d4ac48054ed2e5dfeec04, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/C/0984f06ceb4740c4a5abfd90bf61adae, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/C/0adf4d55f31843248cc219e790d480ce, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/C/d72bce8c83a249a391d08c9d6c3a5a16, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/C/1717d4eff3cf4343bf0e1fd244864b28, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/C/b787863c7ff84cfdb20b1010584fd1ac, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/C/cf89f1fc046d48889e7e586e76b4a23b, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/C/5976489fbca74f4ca633435e391f12e9, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/C/7e9416008d8e4d0785b8177e58000f15, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/C/7f3c94448c75443b9243c01b1efa9241, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/C/14da418ff71c41c69a3febacebd88a8b, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/C/f4795edb7d7d4d3aac22ad9a99bdc211, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/C/3d56fd5c4525452e988fc2410ca6af09, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/C/52e41293bba9416cbf56a974c13d0db5, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/C/b253421c9ea04af0933667f51d1d8fa0, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/C/a807a8475d49448ea8dba33fc8ab598b, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/C/9dc03e419bf642f4ae97b52c9b44ef80, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/C/1f2310154add4781b73f380f5a9a8949, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/C/28e204f9e47f4c5abfb9bd20a16324da, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/C/9f7e78b544d540e697df8bd15a1f1573, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/C/5353308665c24f2d8d35515f770d600e, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/C/e0682746acfa47f49d5fd48aa5cd6e37, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/C/b686eca7a3dc42439d8000f56b8592bb, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/C/17eef9ca1bc04f7f8e07b165c7772b52, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/C/fa1461f5b80647259b51cb623690e569, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/C/38f2c1f13fc14be08a53bc2a04e8cb45, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/C/1b7876c4bf46450e859584468310e960, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/C/c76426a2729047d989b0b20cdd46b39c, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/C/c149193146bd47e0910f36ae9e1a0b21, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/C/77de08181e874b69a367c28e023526bc, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/C/0f8c5fb814544febb5bbf68eccd59528, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/C/a41059273f5d48348460c9f2aae7ed4a, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/C/e20e89b785404b89b92c3d112d8d44c0, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/C/999b744968634309b19688352170de24, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/C/ecf818a0d2c6456697925e8a409e692d, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/C/f753f0f3747a4467b70ea034356954c7, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/C/c395859588b643cc94e29780bf1d7651, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/C/8811c4ec139b412d86d2fda462c0c724, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/C/0d0f26bb74ee45b0aab627371a4fcae5, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/C/f372d8cb7972482bacdf735400486e8a, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/C/585634914bd844f6adc60e42b14d1f61, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/C/7130cada9d3f459dad331b68554df4ec, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/C/4ddac710cbe442149c9b1d17a929ead2] to archive 2023-04-16 00:56:10,727 DEBUG [StoreCloser-TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-16 00:56:10,729 DEBUG [StoreCloser-TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/C/a73b04d521c54866abefb4e0094431e3 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/C/a73b04d521c54866abefb4e0094431e3 2023-04-16 00:56:10,729 DEBUG [StoreCloser-TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/C/330a03ff6a924906b546600756cc25ea to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/C/330a03ff6a924906b546600756cc25ea 2023-04-16 00:56:10,730 DEBUG [StoreCloser-TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/C/eae239871d0845019a296960b5d4fd40 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/C/eae239871d0845019a296960b5d4fd40 2023-04-16 00:56:10,731 DEBUG [StoreCloser-TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/C/a3847ebac68d4ac48054ed2e5dfeec04 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/C/a3847ebac68d4ac48054ed2e5dfeec04 2023-04-16 00:56:10,732 DEBUG [StoreCloser-TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/C/0984f06ceb4740c4a5abfd90bf61adae to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/C/0984f06ceb4740c4a5abfd90bf61adae 2023-04-16 00:56:10,733 DEBUG [StoreCloser-TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/C/0adf4d55f31843248cc219e790d480ce to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/C/0adf4d55f31843248cc219e790d480ce 2023-04-16 00:56:10,734 DEBUG [StoreCloser-TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/C/d72bce8c83a249a391d08c9d6c3a5a16 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/C/d72bce8c83a249a391d08c9d6c3a5a16 2023-04-16 00:56:10,734 DEBUG [StoreCloser-TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/C/1717d4eff3cf4343bf0e1fd244864b28 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/C/1717d4eff3cf4343bf0e1fd244864b28 2023-04-16 00:56:10,735 DEBUG [StoreCloser-TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/C/b787863c7ff84cfdb20b1010584fd1ac to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/C/b787863c7ff84cfdb20b1010584fd1ac 2023-04-16 00:56:10,736 DEBUG [StoreCloser-TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/C/cf89f1fc046d48889e7e586e76b4a23b to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/C/cf89f1fc046d48889e7e586e76b4a23b 2023-04-16 00:56:10,737 DEBUG [StoreCloser-TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/C/5976489fbca74f4ca633435e391f12e9 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/C/5976489fbca74f4ca633435e391f12e9 2023-04-16 00:56:10,738 DEBUG [StoreCloser-TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/C/7e9416008d8e4d0785b8177e58000f15 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/C/7e9416008d8e4d0785b8177e58000f15 2023-04-16 00:56:10,738 DEBUG [StoreCloser-TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/C/7f3c94448c75443b9243c01b1efa9241 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/C/7f3c94448c75443b9243c01b1efa9241 2023-04-16 00:56:10,739 DEBUG [StoreCloser-TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/C/14da418ff71c41c69a3febacebd88a8b to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/C/14da418ff71c41c69a3febacebd88a8b 2023-04-16 00:56:10,740 DEBUG [StoreCloser-TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/C/f4795edb7d7d4d3aac22ad9a99bdc211 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/C/f4795edb7d7d4d3aac22ad9a99bdc211 2023-04-16 00:56:10,741 DEBUG [StoreCloser-TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/C/3d56fd5c4525452e988fc2410ca6af09 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/C/3d56fd5c4525452e988fc2410ca6af09 2023-04-16 00:56:10,741 DEBUG [StoreCloser-TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/C/52e41293bba9416cbf56a974c13d0db5 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/C/52e41293bba9416cbf56a974c13d0db5 2023-04-16 00:56:10,742 DEBUG [StoreCloser-TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/C/b253421c9ea04af0933667f51d1d8fa0 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/C/b253421c9ea04af0933667f51d1d8fa0 2023-04-16 00:56:10,743 DEBUG [StoreCloser-TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/C/a807a8475d49448ea8dba33fc8ab598b to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/C/a807a8475d49448ea8dba33fc8ab598b 2023-04-16 00:56:10,744 DEBUG [StoreCloser-TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/C/9dc03e419bf642f4ae97b52c9b44ef80 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/C/9dc03e419bf642f4ae97b52c9b44ef80 2023-04-16 00:56:10,744 DEBUG [StoreCloser-TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/C/1f2310154add4781b73f380f5a9a8949 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/C/1f2310154add4781b73f380f5a9a8949 2023-04-16 00:56:10,745 DEBUG [StoreCloser-TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/C/28e204f9e47f4c5abfb9bd20a16324da to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/C/28e204f9e47f4c5abfb9bd20a16324da 2023-04-16 00:56:10,746 DEBUG [StoreCloser-TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/C/9f7e78b544d540e697df8bd15a1f1573 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/C/9f7e78b544d540e697df8bd15a1f1573 2023-04-16 00:56:10,747 DEBUG [StoreCloser-TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/C/5353308665c24f2d8d35515f770d600e to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/C/5353308665c24f2d8d35515f770d600e 2023-04-16 00:56:10,747 DEBUG [StoreCloser-TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/C/e0682746acfa47f49d5fd48aa5cd6e37 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/C/e0682746acfa47f49d5fd48aa5cd6e37 2023-04-16 00:56:10,748 DEBUG [StoreCloser-TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/C/b686eca7a3dc42439d8000f56b8592bb to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/C/b686eca7a3dc42439d8000f56b8592bb 2023-04-16 00:56:10,749 DEBUG [StoreCloser-TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/C/17eef9ca1bc04f7f8e07b165c7772b52 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/C/17eef9ca1bc04f7f8e07b165c7772b52 2023-04-16 00:56:10,750 DEBUG [StoreCloser-TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/C/fa1461f5b80647259b51cb623690e569 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/C/fa1461f5b80647259b51cb623690e569 2023-04-16 00:56:10,750 DEBUG [StoreCloser-TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/C/38f2c1f13fc14be08a53bc2a04e8cb45 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/C/38f2c1f13fc14be08a53bc2a04e8cb45 2023-04-16 00:56:10,751 DEBUG [StoreCloser-TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/C/1b7876c4bf46450e859584468310e960 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/C/1b7876c4bf46450e859584468310e960 2023-04-16 00:56:10,752 DEBUG [StoreCloser-TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/C/c76426a2729047d989b0b20cdd46b39c to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/C/c76426a2729047d989b0b20cdd46b39c 2023-04-16 00:56:10,753 DEBUG [StoreCloser-TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/C/c149193146bd47e0910f36ae9e1a0b21 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/C/c149193146bd47e0910f36ae9e1a0b21 2023-04-16 00:56:10,754 DEBUG [StoreCloser-TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/C/77de08181e874b69a367c28e023526bc to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/C/77de08181e874b69a367c28e023526bc 2023-04-16 00:56:10,754 DEBUG [StoreCloser-TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/C/0f8c5fb814544febb5bbf68eccd59528 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/C/0f8c5fb814544febb5bbf68eccd59528 2023-04-16 00:56:10,755 DEBUG [StoreCloser-TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/C/a41059273f5d48348460c9f2aae7ed4a to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/C/a41059273f5d48348460c9f2aae7ed4a 2023-04-16 00:56:10,756 DEBUG [StoreCloser-TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/C/e20e89b785404b89b92c3d112d8d44c0 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/C/e20e89b785404b89b92c3d112d8d44c0 2023-04-16 00:56:10,757 DEBUG [StoreCloser-TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/C/999b744968634309b19688352170de24 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/C/999b744968634309b19688352170de24 2023-04-16 00:56:10,758 DEBUG [StoreCloser-TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/C/ecf818a0d2c6456697925e8a409e692d to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/C/ecf818a0d2c6456697925e8a409e692d 2023-04-16 00:56:10,759 DEBUG [StoreCloser-TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/C/f753f0f3747a4467b70ea034356954c7 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/C/f753f0f3747a4467b70ea034356954c7 2023-04-16 00:56:10,759 DEBUG [StoreCloser-TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/C/c395859588b643cc94e29780bf1d7651 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/C/c395859588b643cc94e29780bf1d7651 2023-04-16 00:56:10,760 DEBUG [StoreCloser-TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/C/8811c4ec139b412d86d2fda462c0c724 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/C/8811c4ec139b412d86d2fda462c0c724 2023-04-16 00:56:10,761 DEBUG [StoreCloser-TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/C/0d0f26bb74ee45b0aab627371a4fcae5 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/C/0d0f26bb74ee45b0aab627371a4fcae5 2023-04-16 00:56:10,762 DEBUG [StoreCloser-TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/C/f372d8cb7972482bacdf735400486e8a to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/C/f372d8cb7972482bacdf735400486e8a 2023-04-16 00:56:10,762 DEBUG [StoreCloser-TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/C/585634914bd844f6adc60e42b14d1f61 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/C/585634914bd844f6adc60e42b14d1f61 2023-04-16 00:56:10,763 DEBUG [StoreCloser-TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/C/7130cada9d3f459dad331b68554df4ec to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/C/7130cada9d3f459dad331b68554df4ec 2023-04-16 00:56:10,764 DEBUG [StoreCloser-TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/C/4ddac710cbe442149c9b1d17a929ead2 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/C/4ddac710cbe442149c9b1d17a929ead2 2023-04-16 00:56:10,768 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/recovered.edits/921.seqid, newMaxSeqId=921, maxSeqId=1 2023-04-16 00:56:10,769 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c. 2023-04-16 00:56:10,769 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 0abd14932d4ec4e5fdc6177f2c7cc55c: 2023-04-16 00:56:10,770 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 0abd14932d4ec4e5fdc6177f2c7cc55c 2023-04-16 00:56:10,771 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=51 updating hbase:meta row=0abd14932d4ec4e5fdc6177f2c7cc55c, regionState=CLOSED 2023-04-16 00:56:10,771 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681606570771"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681606570771"}]},"ts":"1681606570771"} 2023-04-16 00:56:10,773 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=52, resume processing ppid=51 2023-04-16 00:56:10,773 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=52, ppid=51, state=SUCCESS; CloseRegionProcedure 0abd14932d4ec4e5fdc6177f2c7cc55c, server=jenkins-hbase4.apache.org,34975,1681606419217 in 1.5530 sec 2023-04-16 00:56:10,774 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=51, resume processing ppid=50 2023-04-16 00:56:10,774 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=51, ppid=50, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=0abd14932d4ec4e5fdc6177f2c7cc55c, UNASSIGN in 1.5560 sec 2023-04-16 00:56:10,774 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681606570774"}]},"ts":"1681606570774"} 2023-04-16 00:56:10,775 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-16 00:56:10,776 INFO [PEWorker-4] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-16 00:56:10,777 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=50, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 1.5670 sec 2023-04-16 00:56:11,317 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-16 00:56:11,317 INFO [Listener at localhost/33477] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 50 completed 2023-04-16 00:56:11,318 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete TestAcidGuarantees 2023-04-16 00:56:11,318 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure2.ProcedureExecutor(1029): Stored pid=53, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-16 00:56:11,320 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=53, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-16 00:56:11,320 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1227): Checking to see if procedure is done pid=53 2023-04-16 00:56:11,320 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=53, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-16 00:56:11,324 DEBUG [HFileArchiver-11] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/.tmp/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c 2023-04-16 00:56:11,325 DEBUG [HFileArchiver-11] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/.tmp/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/A, FileablePath, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/.tmp/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/B, FileablePath, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/.tmp/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/C, FileablePath, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/.tmp/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/recovered.edits] 2023-04-16 00:56:11,328 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/.tmp/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/A/5322b49c1d9e4da597eb293b436af71f to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/A/5322b49c1d9e4da597eb293b436af71f 2023-04-16 00:56:11,329 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/.tmp/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/A/790f2d4a93c046c9b89f374b29f440f5 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/A/790f2d4a93c046c9b89f374b29f440f5 2023-04-16 00:56:11,330 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/.tmp/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/A/8e9c0baefe0d4dfb8990941574b8da20 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/A/8e9c0baefe0d4dfb8990941574b8da20 2023-04-16 00:56:11,330 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/.tmp/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/A/f43ab8c331144c7e97553c84937b8941 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/A/f43ab8c331144c7e97553c84937b8941 2023-04-16 00:56:11,332 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/.tmp/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/B/1c2fa7f9783c42f1acc17dfee2c1511e to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/B/1c2fa7f9783c42f1acc17dfee2c1511e 2023-04-16 00:56:11,333 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/.tmp/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/B/8f2661f0a2654b90b609ea21bb1144ef to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/B/8f2661f0a2654b90b609ea21bb1144ef 2023-04-16 00:56:11,334 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/.tmp/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/B/e62c19adf57944aeae231befa481ee3e to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/B/e62c19adf57944aeae231befa481ee3e 2023-04-16 00:56:11,335 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/.tmp/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/B/eadeb1fca1a64eec954fc915834d8303 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/B/eadeb1fca1a64eec954fc915834d8303 2023-04-16 00:56:11,337 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/.tmp/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/C/0a85397b09a04b3ca704b9c7de410d79 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/C/0a85397b09a04b3ca704b9c7de410d79 2023-04-16 00:56:11,338 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/.tmp/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/C/0cb89dc6fcda43c8bdedb5d616e65663 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/C/0cb89dc6fcda43c8bdedb5d616e65663 2023-04-16 00:56:11,338 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/.tmp/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/C/9062c7446c1547379c655275d9bcc2d8 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/C/9062c7446c1547379c655275d9bcc2d8 2023-04-16 00:56:11,339 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/.tmp/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/C/abe08fc104fd41c0a091f0b70e27dce8 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/C/abe08fc104fd41c0a091f0b70e27dce8 2023-04-16 00:56:11,341 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/.tmp/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/recovered.edits/921.seqid to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c/recovered.edits/921.seqid 2023-04-16 00:56:11,342 DEBUG [HFileArchiver-11] backup.HFileArchiver(596): Deleted hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/.tmp/data/default/TestAcidGuarantees/0abd14932d4ec4e5fdc6177f2c7cc55c 2023-04-16 00:56:11,342 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-16 00:56:11,344 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=53, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-16 00:56:11,345 WARN [PEWorker-5] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-16 00:56:11,346 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-16 00:56:11,347 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=53, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-16 00:56:11,347 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-16 00:56:11,347 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1681606571347"}]},"ts":"9223372036854775807"} 2023-04-16 00:56:11,348 INFO [PEWorker-5] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-16 00:56:11,348 DEBUG [PEWorker-5] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 0abd14932d4ec4e5fdc6177f2c7cc55c, NAME => 'TestAcidGuarantees,,1681606542191.0abd14932d4ec4e5fdc6177f2c7cc55c.', STARTKEY => '', ENDKEY => ''}] 2023-04-16 00:56:11,348 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-16 00:56:11,348 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1681606571348"}]},"ts":"9223372036854775807"} 2023-04-16 00:56:11,349 INFO [PEWorker-5] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-16 00:56:11,351 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(130): Finished pid=53, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-16 00:56:11,352 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=53, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 33 msec 2023-04-16 00:56:11,421 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1227): Checking to see if procedure is done pid=53 2023-04-16 00:56:11,421 INFO [Listener at localhost/33477] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 53 completed 2023-04-16 00:56:11,433 INFO [Listener at localhost/33477] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testScanAtomicity Thread=227 (was 228), OpenFileDescriptor=622 (was 568) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=419 (was 396) - SystemLoadAverage LEAK? -, ProcessCount=171 (was 171), AvailableMemoryMB=1030 (was 1438) 2023-04-16 00:56:11,444 INFO [Listener at localhost/33477] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testMobGetAtomicity Thread=227, OpenFileDescriptor=622, MaxFileDescriptor=60000, SystemLoadAverage=419, ProcessCount=171, AvailableMemoryMB=1029 2023-04-16 00:56:11,446 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-16 00:56:11,446 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-16 00:56:11,447 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure2.ProcedureExecutor(1029): Stored pid=54, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-16 00:56:11,449 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-16 00:56:11,449 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 54 2023-04-16 00:56:11,449 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-16 00:56:11,449 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-04-16 00:56:11,450 DEBUG [HFileArchiver-12] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/.tmp/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec 2023-04-16 00:56:11,451 DEBUG [HFileArchiver-12] backup.HFileArchiver(153): Directory hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/.tmp/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec empty. 2023-04-16 00:56:11,451 DEBUG [HFileArchiver-12] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/.tmp/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec 2023-04-16 00:56:11,451 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-16 00:56:11,550 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-04-16 00:56:11,751 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-04-16 00:56:11,857 DEBUG [PEWorker-3] util.FSTableDescriptors(570): Wrote into hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-16 00:56:11,858 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => d538367277459b70f4a06235dfaacfec, NAME => 'TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/.tmp 2023-04-16 00:56:12,052 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-04-16 00:56:12,263 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-16 00:56:12,263 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing d538367277459b70f4a06235dfaacfec, disabling compactions & flushes 2023-04-16 00:56:12,263 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. 2023-04-16 00:56:12,263 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. 2023-04-16 00:56:12,263 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. after waiting 0 ms 2023-04-16 00:56:12,263 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. 2023-04-16 00:56:12,263 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. 2023-04-16 00:56:12,263 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for d538367277459b70f4a06235dfaacfec: 2023-04-16 00:56:12,265 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-16 00:56:12,266 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681606572266"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681606572266"}]},"ts":"1681606572266"} 2023-04-16 00:56:12,267 INFO [PEWorker-3] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-16 00:56:12,267 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-16 00:56:12,267 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681606572267"}]},"ts":"1681606572267"} 2023-04-16 00:56:12,268 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-16 00:56:12,274 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=55, ppid=54, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=d538367277459b70f4a06235dfaacfec, ASSIGN}] 2023-04-16 00:56:12,275 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=55, ppid=54, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=d538367277459b70f4a06235dfaacfec, ASSIGN 2023-04-16 00:56:12,276 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=55, ppid=54, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=d538367277459b70f4a06235dfaacfec, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,34975,1681606419217; forceNewPlan=false, retain=false 2023-04-16 00:56:12,427 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=55 updating hbase:meta row=d538367277459b70f4a06235dfaacfec, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:12,427 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681606572427"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681606572427"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681606572427"}]},"ts":"1681606572427"} 2023-04-16 00:56:12,428 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=56, ppid=55, state=RUNNABLE; OpenRegionProcedure d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217}] 2023-04-16 00:56:12,553 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-04-16 00:56:12,582 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. 2023-04-16 00:56:12,582 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => d538367277459b70f4a06235dfaacfec, NAME => 'TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec.', STARTKEY => '', ENDKEY => ''} 2023-04-16 00:56:12,583 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees d538367277459b70f4a06235dfaacfec 2023-04-16 00:56:12,583 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-16 00:56:12,583 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for d538367277459b70f4a06235dfaacfec 2023-04-16 00:56:12,583 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for d538367277459b70f4a06235dfaacfec 2023-04-16 00:56:12,584 INFO [StoreOpener-d538367277459b70f4a06235dfaacfec-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region d538367277459b70f4a06235dfaacfec 2023-04-16 00:56:12,585 DEBUG [StoreOpener-d538367277459b70f4a06235dfaacfec-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A 2023-04-16 00:56:12,585 DEBUG [StoreOpener-d538367277459b70f4a06235dfaacfec-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A 2023-04-16 00:56:12,585 INFO [StoreOpener-d538367277459b70f4a06235dfaacfec-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-16 00:56:12,585 INFO [StoreOpener-d538367277459b70f4a06235dfaacfec-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region d538367277459b70f4a06235dfaacfec columnFamilyName A 2023-04-16 00:56:12,586 INFO [StoreOpener-d538367277459b70f4a06235dfaacfec-1] regionserver.HStore(310): Store=d538367277459b70f4a06235dfaacfec/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-16 00:56:12,586 INFO [StoreOpener-d538367277459b70f4a06235dfaacfec-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region d538367277459b70f4a06235dfaacfec 2023-04-16 00:56:12,587 DEBUG [StoreOpener-d538367277459b70f4a06235dfaacfec-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B 2023-04-16 00:56:12,587 DEBUG [StoreOpener-d538367277459b70f4a06235dfaacfec-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B 2023-04-16 00:56:12,587 INFO [StoreOpener-d538367277459b70f4a06235dfaacfec-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-16 00:56:12,587 INFO [StoreOpener-d538367277459b70f4a06235dfaacfec-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region d538367277459b70f4a06235dfaacfec columnFamilyName B 2023-04-16 00:56:12,588 INFO [StoreOpener-d538367277459b70f4a06235dfaacfec-1] regionserver.HStore(310): Store=d538367277459b70f4a06235dfaacfec/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-16 00:56:12,588 INFO [StoreOpener-d538367277459b70f4a06235dfaacfec-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region d538367277459b70f4a06235dfaacfec 2023-04-16 00:56:12,589 DEBUG [StoreOpener-d538367277459b70f4a06235dfaacfec-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C 2023-04-16 00:56:12,589 DEBUG [StoreOpener-d538367277459b70f4a06235dfaacfec-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C 2023-04-16 00:56:12,589 INFO [StoreOpener-d538367277459b70f4a06235dfaacfec-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-16 00:56:12,589 INFO [StoreOpener-d538367277459b70f4a06235dfaacfec-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region d538367277459b70f4a06235dfaacfec columnFamilyName C 2023-04-16 00:56:12,589 INFO [StoreOpener-d538367277459b70f4a06235dfaacfec-1] regionserver.HStore(310): Store=d538367277459b70f4a06235dfaacfec/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-16 00:56:12,589 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. 2023-04-16 00:56:12,590 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec 2023-04-16 00:56:12,590 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec 2023-04-16 00:56:12,592 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-16 00:56:12,593 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for d538367277459b70f4a06235dfaacfec 2023-04-16 00:56:12,594 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-16 00:56:12,594 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened d538367277459b70f4a06235dfaacfec; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=60816379, jitterRate=-0.09376533329486847}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-16 00:56:12,594 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for d538367277459b70f4a06235dfaacfec: 2023-04-16 00:56:12,595 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec., pid=56, masterSystemTime=1681606572579 2023-04-16 00:56:12,596 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. 2023-04-16 00:56:12,596 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. 2023-04-16 00:56:12,596 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=55 updating hbase:meta row=d538367277459b70f4a06235dfaacfec, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:12,597 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681606572596"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1681606572596"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1681606572596"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1681606572596"}]},"ts":"1681606572596"} 2023-04-16 00:56:12,598 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=56, resume processing ppid=55 2023-04-16 00:56:12,598 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=56, ppid=55, state=SUCCESS; OpenRegionProcedure d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 in 169 msec 2023-04-16 00:56:12,599 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=55, resume processing ppid=54 2023-04-16 00:56:12,599 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=55, ppid=54, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=d538367277459b70f4a06235dfaacfec, ASSIGN in 324 msec 2023-04-16 00:56:12,600 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-16 00:56:12,600 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681606572600"}]},"ts":"1681606572600"} 2023-04-16 00:56:12,600 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-16 00:56:12,602 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-16 00:56:12,603 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=54, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 1.1550 sec 2023-04-16 00:56:13,554 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-04-16 00:56:13,554 INFO [Listener at localhost/33477] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 54 completed 2023-04-16 00:56:13,555 DEBUG [Listener at localhost/33477] zookeeper.ReadOnlyZKClient(139): Connect 0x50529bf3 to 127.0.0.1:53150 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-16 00:56:13,560 DEBUG [Listener at localhost/33477] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3bafb41f, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-16 00:56:13,561 DEBUG [Listener at localhost/33477] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-16 00:56:13,562 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:44530, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-16 00:56:13,563 DEBUG [Listener at localhost/33477] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-16 00:56:13,564 INFO [RS-EventLoopGroup-1-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:46256, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-16 00:56:13,566 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-16 00:56:13,566 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.HMaster$12(2454): Client=jenkins//172.31.14.131 modify table TestAcidGuarantees from 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} to 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', MOB_THRESHOLD => '4', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-16 00:56:13,567 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure2.ProcedureExecutor(1029): Stored pid=57, state=RUNNABLE:MODIFY_TABLE_PREPARE; ModifyTableProcedure table=TestAcidGuarantees 2023-04-16 00:56:13,975 DEBUG [PEWorker-2] util.FSTableDescriptors(570): Wrote into hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-16 00:56:13,976 DEBUG [PEWorker-2] util.FSTableDescriptors(519): Deleted hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-16 00:56:13,976 INFO [PEWorker-2] util.FSTableDescriptors(293): Updated tableinfo=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-16 00:56:13,978 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=58, ppid=57, state=RUNNABLE:REOPEN_TABLE_REGIONS_GET_REGIONS; ReopenTableRegionsProcedure table=TestAcidGuarantees}] 2023-04-16 00:56:13,980 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=59, ppid=58, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=d538367277459b70f4a06235dfaacfec, REOPEN/MOVE}] 2023-04-16 00:56:13,980 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=59, ppid=58, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=d538367277459b70f4a06235dfaacfec, REOPEN/MOVE 2023-04-16 00:56:13,981 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=59 updating hbase:meta row=d538367277459b70f4a06235dfaacfec, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:13,981 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681606573981"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681606573981"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681606573981"}]},"ts":"1681606573981"} 2023-04-16 00:56:13,982 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=60, ppid=59, state=RUNNABLE; CloseRegionProcedure d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217}] 2023-04-16 00:56:14,133 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close d538367277459b70f4a06235dfaacfec 2023-04-16 00:56:14,134 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing d538367277459b70f4a06235dfaacfec, disabling compactions & flushes 2023-04-16 00:56:14,134 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. 2023-04-16 00:56:14,134 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. 2023-04-16 00:56:14,134 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. after waiting 0 ms 2023-04-16 00:56:14,134 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. 2023-04-16 00:56:14,137 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-04-16 00:56:14,137 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. 2023-04-16 00:56:14,137 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for d538367277459b70f4a06235dfaacfec: 2023-04-16 00:56:14,138 WARN [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(3509): Not adding moved region record: d538367277459b70f4a06235dfaacfec to self. 2023-04-16 00:56:14,139 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed d538367277459b70f4a06235dfaacfec 2023-04-16 00:56:14,139 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=59 updating hbase:meta row=d538367277459b70f4a06235dfaacfec, regionState=CLOSED 2023-04-16 00:56:14,139 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681606574139"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681606574139"}]},"ts":"1681606574139"} 2023-04-16 00:56:14,141 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=60, resume processing ppid=59 2023-04-16 00:56:14,141 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=60, ppid=59, state=SUCCESS; CloseRegionProcedure d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 in 158 msec 2023-04-16 00:56:14,141 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=59, ppid=58, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=d538367277459b70f4a06235dfaacfec, REOPEN/MOVE; state=CLOSED, location=jenkins-hbase4.apache.org,34975,1681606419217; forceNewPlan=false, retain=true 2023-04-16 00:56:14,292 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=59 updating hbase:meta row=d538367277459b70f4a06235dfaacfec, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:14,292 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681606574292"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681606574292"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681606574292"}]},"ts":"1681606574292"} 2023-04-16 00:56:14,293 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=61, ppid=59, state=RUNNABLE; OpenRegionProcedure d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217}] 2023-04-16 00:56:14,447 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. 2023-04-16 00:56:14,448 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => d538367277459b70f4a06235dfaacfec, NAME => 'TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec.', STARTKEY => '', ENDKEY => ''} 2023-04-16 00:56:14,448 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees d538367277459b70f4a06235dfaacfec 2023-04-16 00:56:14,448 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-16 00:56:14,448 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for d538367277459b70f4a06235dfaacfec 2023-04-16 00:56:14,448 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for d538367277459b70f4a06235dfaacfec 2023-04-16 00:56:14,449 INFO [StoreOpener-d538367277459b70f4a06235dfaacfec-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region d538367277459b70f4a06235dfaacfec 2023-04-16 00:56:14,450 DEBUG [StoreOpener-d538367277459b70f4a06235dfaacfec-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A 2023-04-16 00:56:14,450 DEBUG [StoreOpener-d538367277459b70f4a06235dfaacfec-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A 2023-04-16 00:56:14,450 INFO [StoreOpener-d538367277459b70f4a06235dfaacfec-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-16 00:56:14,450 INFO [StoreOpener-d538367277459b70f4a06235dfaacfec-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region d538367277459b70f4a06235dfaacfec columnFamilyName A 2023-04-16 00:56:14,451 INFO [StoreOpener-d538367277459b70f4a06235dfaacfec-1] regionserver.HStore(310): Store=d538367277459b70f4a06235dfaacfec/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-16 00:56:14,452 INFO [StoreOpener-d538367277459b70f4a06235dfaacfec-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region d538367277459b70f4a06235dfaacfec 2023-04-16 00:56:14,452 DEBUG [StoreOpener-d538367277459b70f4a06235dfaacfec-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B 2023-04-16 00:56:14,452 DEBUG [StoreOpener-d538367277459b70f4a06235dfaacfec-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B 2023-04-16 00:56:14,452 INFO [StoreOpener-d538367277459b70f4a06235dfaacfec-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-16 00:56:14,452 INFO [StoreOpener-d538367277459b70f4a06235dfaacfec-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region d538367277459b70f4a06235dfaacfec columnFamilyName B 2023-04-16 00:56:14,453 INFO [StoreOpener-d538367277459b70f4a06235dfaacfec-1] regionserver.HStore(310): Store=d538367277459b70f4a06235dfaacfec/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-16 00:56:14,453 INFO [StoreOpener-d538367277459b70f4a06235dfaacfec-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region d538367277459b70f4a06235dfaacfec 2023-04-16 00:56:14,454 DEBUG [StoreOpener-d538367277459b70f4a06235dfaacfec-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C 2023-04-16 00:56:14,454 DEBUG [StoreOpener-d538367277459b70f4a06235dfaacfec-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C 2023-04-16 00:56:14,454 INFO [StoreOpener-d538367277459b70f4a06235dfaacfec-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-16 00:56:14,454 INFO [StoreOpener-d538367277459b70f4a06235dfaacfec-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region d538367277459b70f4a06235dfaacfec columnFamilyName C 2023-04-16 00:56:14,454 INFO [StoreOpener-d538367277459b70f4a06235dfaacfec-1] regionserver.HStore(310): Store=d538367277459b70f4a06235dfaacfec/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-16 00:56:14,454 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. 2023-04-16 00:56:14,455 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec 2023-04-16 00:56:14,456 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec 2023-04-16 00:56:14,457 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-16 00:56:14,458 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for d538367277459b70f4a06235dfaacfec 2023-04-16 00:56:14,459 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened d538367277459b70f4a06235dfaacfec; next sequenceid=5; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=70203272, jitterRate=0.04611027240753174}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-16 00:56:14,459 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for d538367277459b70f4a06235dfaacfec: 2023-04-16 00:56:14,459 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec., pid=61, masterSystemTime=1681606574445 2023-04-16 00:56:14,460 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. 2023-04-16 00:56:14,460 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. 2023-04-16 00:56:14,461 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=59 updating hbase:meta row=d538367277459b70f4a06235dfaacfec, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:14,461 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681606574461"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1681606574461"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1681606574461"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1681606574461"}]},"ts":"1681606574461"} 2023-04-16 00:56:14,462 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=61, resume processing ppid=59 2023-04-16 00:56:14,462 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=61, ppid=59, state=SUCCESS; OpenRegionProcedure d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 in 169 msec 2023-04-16 00:56:14,463 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=59, resume processing ppid=58 2023-04-16 00:56:14,463 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=59, ppid=58, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=d538367277459b70f4a06235dfaacfec, REOPEN/MOVE in 482 msec 2023-04-16 00:56:14,465 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=58, resume processing ppid=57 2023-04-16 00:56:14,466 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=58, ppid=57, state=SUCCESS; ReopenTableRegionsProcedure table=TestAcidGuarantees in 486 msec 2023-04-16 00:56:14,467 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=57, state=SUCCESS; ModifyTableProcedure table=TestAcidGuarantees in 900 msec 2023-04-16 00:56:14,467 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1227): Checking to see if procedure is done pid=57 2023-04-16 00:56:14,468 DEBUG [Listener at localhost/33477] zookeeper.ReadOnlyZKClient(139): Connect 0x470742aa to 127.0.0.1:53150 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-16 00:56:14,473 DEBUG [Listener at localhost/33477] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3ced4ba, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-16 00:56:14,473 DEBUG [Listener at localhost/33477] zookeeper.ReadOnlyZKClient(139): Connect 0x1e3503b4 to 127.0.0.1:53150 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-16 00:56:14,478 DEBUG [Listener at localhost/33477] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@63da10ca, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-16 00:56:14,478 DEBUG [Listener at localhost/33477] zookeeper.ReadOnlyZKClient(139): Connect 0x6f06dd50 to 127.0.0.1:53150 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-16 00:56:14,482 DEBUG [Listener at localhost/33477] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@59185ba7, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-16 00:56:14,482 DEBUG [Listener at localhost/33477] zookeeper.ReadOnlyZKClient(139): Connect 0x5f9e8654 to 127.0.0.1:53150 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-16 00:56:14,485 DEBUG [Listener at localhost/33477] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@62b3caa, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-16 00:56:14,485 DEBUG [Listener at localhost/33477] zookeeper.ReadOnlyZKClient(139): Connect 0x4db22237 to 127.0.0.1:53150 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-16 00:56:14,489 DEBUG [Listener at localhost/33477] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@66b5b1f3, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-16 00:56:14,489 DEBUG [Listener at localhost/33477] zookeeper.ReadOnlyZKClient(139): Connect 0x544a4215 to 127.0.0.1:53150 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-16 00:56:14,496 DEBUG [Listener at localhost/33477] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1fbb5b46, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-16 00:56:14,496 DEBUG [Listener at localhost/33477] zookeeper.ReadOnlyZKClient(139): Connect 0x25baee3e to 127.0.0.1:53150 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-16 00:56:14,503 DEBUG [Listener at localhost/33477] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@23e65a76, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-16 00:56:14,503 DEBUG [Listener at localhost/33477] zookeeper.ReadOnlyZKClient(139): Connect 0x2c53bf96 to 127.0.0.1:53150 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-16 00:56:14,509 DEBUG [Listener at localhost/33477] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5e6a6f3c, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-16 00:56:14,509 DEBUG [Listener at localhost/33477] zookeeper.ReadOnlyZKClient(139): Connect 0x3b59bdcf to 127.0.0.1:53150 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-16 00:56:14,513 DEBUG [Listener at localhost/33477] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@a470a3d, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-16 00:56:14,513 DEBUG [Listener at localhost/33477] zookeeper.ReadOnlyZKClient(139): Connect 0x3d765193 to 127.0.0.1:53150 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-16 00:56:14,519 DEBUG [Listener at localhost/33477] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4bb72437, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-16 00:56:14,527 DEBUG [hconnection-0x4d48c7f1-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-16 00:56:14,527 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:56:14,528 DEBUG [hconnection-0x6276695a-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-16 00:56:14,528 DEBUG [hconnection-0x41701318-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-16 00:56:14,529 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:44550, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-16 00:56:14,533 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:44548, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-16 00:56:14,533 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:44542, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-16 00:56:14,533 DEBUG [hconnection-0x7115f0e5-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-16 00:56:14,533 DEBUG [hconnection-0x42be3036-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-16 00:56:14,537 DEBUG [hconnection-0x9945ba6-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-16 00:56:14,537 DEBUG [hconnection-0x2e065648-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-16 00:56:14,537 DEBUG [hconnection-0x20a8931f-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-16 00:56:14,538 DEBUG [hconnection-0x69a5f29f-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-16 00:56:14,538 DEBUG [hconnection-0x22cf1f2a-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-16 00:56:14,538 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:44562, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-16 00:56:14,539 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:44590, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-16 00:56:14,544 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:44556, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-16 00:56:14,544 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:44576, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-16 00:56:14,545 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:44554, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-16 00:56:14,545 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:44564, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-16 00:56:14,546 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:44600, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-16 00:56:14,549 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:56:14,550 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:56:14,551 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:56:14,551 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:56:14,551 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:56:14,551 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:56:14,551 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:14,552 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:14,553 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:14,553 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:56:14,553 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:56:14,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:56:14,553 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:14,553 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:56:14,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:14,553 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:14,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:56:14,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:14,554 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:56:14,554 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(9158): Flush requested on d538367277459b70f4a06235dfaacfec 2023-04-16 00:56:14,554 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing d538367277459b70f4a06235dfaacfec 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-16 00:56:14,555 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:56:14,555 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:56:14,555 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:56:14,555 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:56:14,555 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:56:14,555 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d538367277459b70f4a06235dfaacfec, store=A 2023-04-16 00:56:14,555 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:56:14,555 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d538367277459b70f4a06235dfaacfec, store=B 2023-04-16 00:56:14,555 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:56:14,555 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d538367277459b70f4a06235dfaacfec, store=C 2023-04-16 00:56:14,555 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:56:14,555 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:56:14,555 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:56:14,555 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. 2023-04-16 00:56:14,555 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. started... 2023-04-16 00:56:14,555 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. as already flushing 2023-04-16 00:56:14,555 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. 2023-04-16 00:56:14,555 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:56:14,555 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:56:14,555 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:14,556 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:56:14,556 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:56:14,557 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:14,557 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:14,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:14,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:14,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:14,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:14,558 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:14,558 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:56:14,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:14,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:14,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:14,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:14,560 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:56:14,560 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5770ec45[Count = 0] remaining members to acquire global barrier 2023-04-16 00:56:14,560 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:56:14,560 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:14,561 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:14,561 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:14,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:14,561 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:56:14,561 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:56:14,561 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:56:14,564 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:56:14,564 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:56:14,564 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:56:14,564 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:14,564 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:56:14,564 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:56:14,564 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:56:14,564 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:56:14,564 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:56:14,564 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:56:14,564 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:56:14,566 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:14,566 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:56:14,566 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:14,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:14,566 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:14,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:14,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:14,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:14,566 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:56:14,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:56:14,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:14,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:14,567 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:14,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:14,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:14,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:14,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:14,569 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:14,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:14,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:14,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:14,571 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:56:14,571 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:14,571 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:56:14,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:56:14,571 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:56:14,571 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:56:14,571 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:56:14,571 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:56:14,571 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:56:14,571 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:56:14,571 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:14,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:56:14,571 DEBUG [Thread-4138] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:56:14,571 DEBUG [Thread-4138] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:56:14,571 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:14,572 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:14,572 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:14,572 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:14,572 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44562 deadline: 1681606634568, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:14,572 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:14,572 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:14,572 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:14,572 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44564 deadline: 1681606634568, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:14,573 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681606574566, max timestamp=1681606574571Num uniques 859; 2023-04-16 00:56:14,573 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-16 00:56:14,573 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681606574566, max timestamp=1681606574571Num uniques 859; 2023-04-16 00:56:14,573 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-16 00:56:14,574 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681606574566, max timestamp=1681606574571Num uniques 859; 2023-04-16 00:56:14,574 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-16 00:56:14,578 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304169baba9bf33654eaf950b946fa590ce3e to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304169baba9bf33654eaf950b946fa590ce3e 2023-04-16 00:56:14,584 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=16, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp/A/abd5cdebfeef4d6e8e910c771a0c80aa 2023-04-16 00:56:14,606 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:14,606 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44548 deadline: 1681606634603, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:14,606 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:14,606 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44556 deadline: 1681606634603, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:14,606 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:14,607 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44554 deadline: 1681606634604, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:14,672 DEBUG [Thread-4138] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:56:14,672 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:56:14,674 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:56:14,674 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:14,675 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44562 deadline: 1681606634673, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:14,675 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:14,675 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44564 deadline: 1681606634673, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:14,675 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:56:14,675 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:56:14,675 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:56:14,675 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:56:14,675 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:56:14,675 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:56:14,676 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:14,676 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:14,677 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:56:14,677 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:14,677 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:14,677 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:56:14,678 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:56:14,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:56:14,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:14,678 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:14,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:56:14,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:14,679 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:56:14,679 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:56:14,679 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:56:14,679 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:56:14,679 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:56:14,679 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:56:14,680 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:56:14,680 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. 2023-04-16 00:56:14,680 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. started... 2023-04-16 00:56:14,680 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. as already flushing 2023-04-16 00:56:14,680 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. 2023-04-16 00:56:14,680 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:56:14,680 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:56:14,680 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:14,680 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:56:14,680 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:56:14,682 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:14,682 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:14,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:14,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:14,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:14,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:14,683 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:14,683 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:56:14,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:14,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:14,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:14,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:14,684 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:56:14,684 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6f2ca587[Count = 0] remaining members to acquire global barrier 2023-04-16 00:56:14,684 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:56:14,684 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:14,685 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:14,685 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:14,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:14,685 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:56:14,685 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:56:14,685 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:56:14,685 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:14,685 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:56:14,686 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:56:14,686 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:56:14,687 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:56:14,687 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:14,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:14,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:14,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:14,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:14,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:14,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:14,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:14,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:14,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:14,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:14,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:56:14,689 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:56:14,689 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:56:14,689 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:56:14,689 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:56:14,689 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:56:14,689 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:56:14,698 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:14,698 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:14,698 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:14,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:14,698 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:56:14,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:14,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:14,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:14,699 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:56:14,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:56:14,699 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:14,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:14,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:14,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:14,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:14,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:14,700 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:14,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:14,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:14,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:14,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:14,704 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:14,704 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:14,704 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:14,704 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:14,704 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:14,704 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:56:14,704 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:56:14,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:56:14,704 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:56:14,704 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:14,704 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:56:14,704 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:56:14,704 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:56:14,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:56:14,704 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:56:14,704 DEBUG [Thread-4138] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:56:14,704 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:56:14,704 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:14,704 DEBUG [Thread-4138] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:56:14,708 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:14,708 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44556 deadline: 1681606634707, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:14,709 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:14,709 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44548 deadline: 1681606634707, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:14,709 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:14,709 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44554 deadline: 1681606634707, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:14,805 DEBUG [Thread-4138] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:56:14,805 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:56:14,807 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:56:14,808 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:56:14,808 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:56:14,808 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:56:14,808 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:56:14,808 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:56:14,808 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:56:14,809 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:14,809 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:14,810 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:14,810 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:56:14,810 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:56:14,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:56:14,810 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:14,810 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:56:14,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:14,811 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:14,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:56:14,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:14,811 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:56:14,811 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:56:14,811 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:56:14,811 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:56:14,811 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:56:14,811 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:56:14,811 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:56:14,811 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:56:14,811 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. 2023-04-16 00:56:14,812 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. started... 2023-04-16 00:56:14,812 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. as already flushing 2023-04-16 00:56:14,812 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. 2023-04-16 00:56:14,812 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:56:14,812 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:56:14,812 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:14,812 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:56:14,812 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:56:14,813 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:14,813 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:14,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:14,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:14,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:14,813 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:14,814 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:56:14,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:14,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:14,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:14,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:14,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:14,815 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:56:14,815 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4f53105f[Count = 0] remaining members to acquire global barrier 2023-04-16 00:56:14,815 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:56:14,815 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:14,816 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:14,816 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:14,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:14,816 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:56:14,816 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:56:14,816 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:56:14,816 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:14,816 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:56:14,819 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:14,819 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:56:14,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:14,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:14,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:14,819 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:56:14,820 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:56:14,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:14,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:14,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:14,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:14,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:14,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:14,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:14,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:56:14,822 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:56:14,822 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:56:14,822 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:56:14,822 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:56:14,822 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:56:14,822 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:56:14,824 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:14,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:14,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:14,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:14,824 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:14,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:14,824 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:14,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:14,824 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:56:14,824 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:14,825 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:56:14,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:56:14,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:14,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:14,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:14,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:14,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:14,827 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:14,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:14,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:14,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:14,829 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:14,829 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:56:14,829 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:14,829 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:56:14,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:56:14,829 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:56:14,829 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:56:14,829 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:56:14,829 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:56:14,829 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:14,829 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:14,829 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:14,829 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:14,829 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:56:14,829 DEBUG [Thread-4138] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:56:14,829 DEBUG [Thread-4138] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:56:14,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:56:14,829 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:14,877 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:14,877 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44564 deadline: 1681606634876, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:14,878 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:14,879 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44562 deadline: 1681606634876, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:14,911 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:14,911 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44556 deadline: 1681606634909, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:14,912 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:14,912 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44548 deadline: 1681606634910, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:14,912 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:14,912 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44554 deadline: 1681606634911, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:14,929 DEBUG [Thread-4138] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:56:14,930 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:56:14,931 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:56:14,936 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:56:14,936 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:56:14,936 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:56:14,936 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:56:14,936 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:56:14,936 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:56:14,936 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:14,936 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:14,937 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:56:14,937 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:14,937 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:56:14,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:56:14,938 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:14,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:14,938 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:56:14,938 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:14,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:56:14,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:14,939 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:56:14,944 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:56:14,944 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:56:14,944 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:56:14,944 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:56:14,944 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:56:14,948 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:56:14,948 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:56:14,948 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. 2023-04-16 00:56:14,948 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. started... 2023-04-16 00:56:14,948 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. as already flushing 2023-04-16 00:56:14,948 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. 2023-04-16 00:56:14,948 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:56:14,948 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:56:14,948 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:14,948 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:56:14,948 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:56:14,951 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:14,951 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:14,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:14,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:14,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:14,951 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:14,951 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:56:14,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:14,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:14,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:14,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:14,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:14,952 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:56:14,952 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@28398e08[Count = 0] remaining members to acquire global barrier 2023-04-16 00:56:14,952 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:56:14,952 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:14,953 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:14,953 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:14,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:14,953 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:56:14,953 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:56:14,953 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:56:14,954 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:14,954 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:56:14,955 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:14,955 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:56:14,955 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:56:14,955 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:56:14,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:14,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:14,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:14,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:14,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:14,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:14,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:14,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:14,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:14,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:14,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:56:14,957 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:56:14,957 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:56:14,957 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:56:14,957 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:56:14,957 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:56:14,957 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:56:14,959 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:14,959 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:56:14,959 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:14,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:14,959 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:14,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:14,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:14,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:14,959 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:56:14,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:56:14,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:14,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:14,960 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:14,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:14,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:14,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:14,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:14,961 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:14,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:14,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:14,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:14,964 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:14,964 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:14,964 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:14,964 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:14,964 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:14,964 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:56:14,964 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:56:14,964 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:56:14,965 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:56:14,964 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:56:14,965 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:56:14,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:56:14,964 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:14,965 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:14,965 DEBUG [Thread-4138] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:56:14,965 DEBUG [Thread-4138] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:56:14,965 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:56:14,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:56:15,006 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=16 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp/B/df0e86e715234f7aa1b3cb278f41ed4c 2023-04-16 00:56:15,021 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=16 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp/C/124b949b8e2c4623b6d3c107c5e15ac6 2023-04-16 00:56:15,025 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp/A/abd5cdebfeef4d6e8e910c771a0c80aa as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A/abd5cdebfeef4d6e8e910c771a0c80aa 2023-04-16 00:56:15,029 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A/abd5cdebfeef4d6e8e910c771a0c80aa, entries=200, sequenceid=16, filesize=31.9 K 2023-04-16 00:56:15,030 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp/B/df0e86e715234f7aa1b3cb278f41ed4c as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/df0e86e715234f7aa1b3cb278f41ed4c 2023-04-16 00:56:15,033 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/df0e86e715234f7aa1b3cb278f41ed4c, entries=150, sequenceid=16, filesize=11.6 K 2023-04-16 00:56:15,034 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp/C/124b949b8e2c4623b6d3c107c5e15ac6 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/124b949b8e2c4623b6d3c107c5e15ac6 2023-04-16 00:56:15,039 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/124b949b8e2c4623b6d3c107c5e15ac6, entries=150, sequenceid=16, filesize=11.6 K 2023-04-16 00:56:15,040 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=127.47 KB/130530 for d538367277459b70f4a06235dfaacfec in 486ms, sequenceid=16, compaction requested=false 2023-04-16 00:56:15,040 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for d538367277459b70f4a06235dfaacfec: 2023-04-16 00:56:15,065 DEBUG [Thread-4138] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:56:15,066 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:56:15,067 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:56:15,068 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:56:15,068 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:56:15,068 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:56:15,069 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:56:15,069 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:56:15,069 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:56:15,069 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:15,069 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:15,070 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:56:15,070 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:15,070 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:56:15,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:56:15,071 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:15,071 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:56:15,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:15,071 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:15,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:56:15,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:15,071 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:56:15,072 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:56:15,072 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:56:15,072 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:56:15,072 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:56:15,072 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:56:15,072 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:56:15,072 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:56:15,072 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. 2023-04-16 00:56:15,072 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. started... 2023-04-16 00:56:15,073 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing d538367277459b70f4a06235dfaacfec 3/3 column families, dataSize=127.47 KB heapSize=365.06 KB 2023-04-16 00:56:15,073 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d538367277459b70f4a06235dfaacfec, store=A 2023-04-16 00:56:15,073 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 00:56:15,073 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d538367277459b70f4a06235dfaacfec, store=B 2023-04-16 00:56:15,073 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 00:56:15,073 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d538367277459b70f4a06235dfaacfec, store=C 2023-04-16 00:56:15,073 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 00:56:15,089 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304169255505b063b45588475d1b1ccc8285f to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304169255505b063b45588475d1b1ccc8285f 2023-04-16 00:56:15,097 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=52, memsize=42.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp/A/9f993bd920d148539f837ee17dae4620 2023-04-16 00:56:15,114 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=52 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp/B/621929dbbab54631aa24fa3dfc6566b4 2023-04-16 00:56:15,130 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=52 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp/C/edc54e20ad384b238b805bdb755bbfe4 2023-04-16 00:56:15,134 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp/A/9f993bd920d148539f837ee17dae4620 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A/9f993bd920d148539f837ee17dae4620 2023-04-16 00:56:15,137 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A/9f993bd920d148539f837ee17dae4620, entries=150, sequenceid=52, filesize=25.1 K 2023-04-16 00:56:15,138 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp/B/621929dbbab54631aa24fa3dfc6566b4 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/621929dbbab54631aa24fa3dfc6566b4 2023-04-16 00:56:15,142 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/621929dbbab54631aa24fa3dfc6566b4, entries=150, sequenceid=52, filesize=11.6 K 2023-04-16 00:56:15,142 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp/C/edc54e20ad384b238b805bdb755bbfe4 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/edc54e20ad384b238b805bdb755bbfe4 2023-04-16 00:56:15,146 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/edc54e20ad384b238b805bdb755bbfe4, entries=150, sequenceid=52, filesize=11.6 K 2023-04-16 00:56:15,147 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~365.02 KB/373776, currentSize=0 B/0 for d538367277459b70f4a06235dfaacfec in 74ms, sequenceid=52, compaction requested=false 2023-04-16 00:56:15,147 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for d538367277459b70f4a06235dfaacfec: 2023-04-16 00:56:15,147 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. 2023-04-16 00:56:15,147 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:56:15,147 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:56:15,147 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:15,147 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:56:15,147 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:56:15,148 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:15,148 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:15,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:15,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:15,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:15,149 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:15,149 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:56:15,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:15,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:15,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:15,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:15,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:15,150 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:56:15,150 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@78fb5d8e[Count = 0] remaining members to acquire global barrier 2023-04-16 00:56:15,150 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:56:15,150 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:15,151 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:15,152 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:15,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:15,152 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:56:15,152 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:56:15,152 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:15,152 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:56:15,152 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:56:15,154 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:56:15,154 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:56:15,154 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:56:15,154 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:15,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:15,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:15,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:15,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:15,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:15,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:15,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:15,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:15,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:15,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:15,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:56:15,157 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:56:15,157 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:56:15,157 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:56:15,157 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:56:15,157 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:56:15,157 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:56:15,158 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:15,158 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:15,158 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:15,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:15,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:15,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:15,158 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:56:15,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:15,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:15,159 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:15,159 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:56:15,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:56:15,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:15,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:15,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:15,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:15,160 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:15,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:15,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:15,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:15,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:15,165 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:15,165 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:56:15,165 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:15,165 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:56:15,165 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:56:15,165 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:56:15,165 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:56:15,165 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:56:15,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:56:15,165 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:15,165 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:15,165 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:15,165 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:15,165 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:15,165 DEBUG [Thread-4138] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:56:15,165 DEBUG [Thread-4138] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:56:15,165 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:56:15,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:56:15,166 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:56:15,216 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(9158): Flush requested on d538367277459b70f4a06235dfaacfec 2023-04-16 00:56:15,216 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing d538367277459b70f4a06235dfaacfec 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-16 00:56:15,217 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d538367277459b70f4a06235dfaacfec, store=A 2023-04-16 00:56:15,217 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:56:15,217 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d538367277459b70f4a06235dfaacfec, store=B 2023-04-16 00:56:15,217 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:56:15,217 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d538367277459b70f4a06235dfaacfec, store=C 2023-04-16 00:56:15,217 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:56:15,233 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:15,233 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44562 deadline: 1681606635230, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:15,233 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:15,233 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44556 deadline: 1681606635230, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:15,235 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:15,235 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44564 deadline: 1681606635232, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:15,235 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:15,236 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44554 deadline: 1681606635233, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:15,236 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:15,236 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44548 deadline: 1681606635233, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:15,266 DEBUG [Thread-4138] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:56:15,266 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:56:15,268 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:56:15,269 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:56:15,269 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:56:15,270 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:56:15,270 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:56:15,270 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:56:15,270 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:56:15,270 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:15,270 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:15,273 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:56:15,273 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:15,273 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:56:15,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:56:15,273 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:15,273 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:56:15,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:15,274 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:15,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:56:15,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:15,274 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:56:15,274 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:56:15,274 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:56:15,274 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:56:15,274 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:56:15,274 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:56:15,275 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:56:15,275 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:56:15,275 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. 2023-04-16 00:56:15,275 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. started... 2023-04-16 00:56:15,275 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. as already flushing 2023-04-16 00:56:15,275 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. 2023-04-16 00:56:15,275 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:56:15,275 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:56:15,275 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:15,275 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:56:15,275 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:56:15,276 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:15,277 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:15,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:15,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:15,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:15,277 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:15,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:15,277 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:56:15,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:15,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:15,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:15,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:15,279 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:56:15,279 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7271df8b[Count = 0] remaining members to acquire global barrier 2023-04-16 00:56:15,279 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:56:15,279 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:15,280 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:15,280 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:15,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:15,280 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:56:15,280 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:56:15,280 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:15,280 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:56:15,280 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:56:15,282 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:15,282 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:56:15,282 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:56:15,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:15,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:15,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:15,282 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:56:15,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:15,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:15,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:15,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:15,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:15,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:15,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:15,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:56:15,285 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:56:15,285 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:56:15,285 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:56:15,285 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:56:15,285 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:56:15,285 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:56:15,287 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:15,287 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:15,287 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:15,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:15,287 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:56:15,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:15,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:15,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:15,288 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:56:15,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:56:15,288 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:15,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:15,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:15,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:15,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:15,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:15,290 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:15,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:15,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:15,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:15,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:15,292 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:56:15,292 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:56:15,292 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:56:15,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:56:15,293 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:15,293 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:15,293 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:15,293 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:15,293 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:56:15,293 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:56:15,293 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:56:15,293 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:15,293 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:15,293 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:15,293 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:56:15,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:56:15,293 DEBUG [Thread-4138] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:56:15,293 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:56:15,293 DEBUG [Thread-4138] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:56:15,336 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:15,336 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44562 deadline: 1681606635334, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:15,336 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:15,336 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44556 deadline: 1681606635334, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:15,337 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:15,337 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44564 deadline: 1681606635336, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:15,337 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:15,338 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44554 deadline: 1681606635336, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:15,339 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:15,340 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44548 deadline: 1681606635337, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:15,393 DEBUG [Thread-4138] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:56:15,394 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:56:15,395 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:56:15,397 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:56:15,397 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:56:15,397 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:56:15,397 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:56:15,397 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:56:15,397 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:56:15,398 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:15,398 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:15,399 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:56:15,399 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:56:15,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:56:15,399 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:15,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:15,400 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:15,400 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:56:15,400 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:15,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:56:15,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:15,400 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:56:15,404 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:56:15,404 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:56:15,404 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:56:15,404 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:56:15,404 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:56:15,408 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:56:15,408 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:56:15,408 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. 2023-04-16 00:56:15,408 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. started... 2023-04-16 00:56:15,408 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. as already flushing 2023-04-16 00:56:15,408 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. 2023-04-16 00:56:15,408 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:56:15,409 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:56:15,409 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:15,409 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:56:15,409 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:56:15,410 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:15,410 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:15,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:15,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:15,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:15,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:15,411 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:15,411 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:56:15,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:15,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:15,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:15,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:15,412 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:56:15,412 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2577e872[Count = 0] remaining members to acquire global barrier 2023-04-16 00:56:15,412 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:56:15,412 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:15,415 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:15,415 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:15,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:15,415 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:56:15,415 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:56:15,415 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:56:15,415 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:15,415 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:56:15,416 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:15,416 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:56:15,416 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:56:15,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:15,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:15,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:15,416 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:56:15,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:15,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:15,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:15,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:15,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:15,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:15,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:15,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:56:15,419 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:56:15,419 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:56:15,419 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:56:15,419 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:56:15,419 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:56:15,419 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:56:15,421 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:15,421 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:56:15,421 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:15,421 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:15,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:15,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:15,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:15,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:15,421 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:15,421 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:56:15,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:56:15,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:15,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:15,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:15,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:15,422 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:15,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:15,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:15,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:15,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:15,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:15,424 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:15,425 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:15,425 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:15,425 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:56:15,425 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:56:15,425 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:15,425 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:56:15,425 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:15,425 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:15,425 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:15,425 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:56:15,425 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:56:15,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:56:15,425 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:56:15,425 DEBUG [Thread-4138] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:56:15,425 DEBUG [Thread-4138] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:56:15,426 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:56:15,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:56:15,426 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:56:15,526 DEBUG [Thread-4138] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:56:15,526 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:56:15,527 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:56:15,528 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:56:15,529 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:56:15,529 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:56:15,529 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:56:15,529 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:56:15,529 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:56:15,529 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:15,529 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:15,532 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:56:15,532 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:15,532 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:56:15,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:56:15,532 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:15,532 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:56:15,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:15,533 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:15,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:56:15,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:15,533 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:56:15,533 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:56:15,533 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:56:15,533 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:56:15,533 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:56:15,533 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:56:15,534 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:56:15,534 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:56:15,534 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. 2023-04-16 00:56:15,534 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. started... 2023-04-16 00:56:15,534 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. as already flushing 2023-04-16 00:56:15,534 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. 2023-04-16 00:56:15,534 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:56:15,534 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:56:15,534 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:15,534 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:56:15,534 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:56:15,536 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:15,536 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:15,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:15,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:15,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:15,536 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:15,536 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:56:15,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:15,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:15,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:15,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:15,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:15,538 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:56:15,538 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@41da7b1a[Count = 0] remaining members to acquire global barrier 2023-04-16 00:56:15,539 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:56:15,539 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:15,539 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:15,539 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44556 deadline: 1681606635537, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:15,540 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:15,540 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44562 deadline: 1681606635538, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:15,540 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:15,540 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:15,540 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44564 deadline: 1681606635539, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:15,540 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:15,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:15,541 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:56:15,541 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:56:15,541 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:56:15,542 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:15,542 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:56:15,542 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:15,542 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44554 deadline: 1681606635540, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:15,542 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:56:15,542 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:56:15,542 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:15,542 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:56:15,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:15,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:15,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:15,543 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:15,543 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44548 deadline: 1681606635541, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:15,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:15,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:15,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:15,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:15,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:15,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:15,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:15,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:56:15,545 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:56:15,545 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:56:15,545 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:56:15,545 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:56:15,545 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:56:15,546 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:56:15,548 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:15,548 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:15,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:15,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:15,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:15,548 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:15,548 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:56:15,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:15,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:15,549 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:56:15,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:56:15,549 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:15,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:15,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:15,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:15,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:15,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:15,550 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:15,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:15,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:15,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:15,555 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:15,555 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:15,555 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:15,555 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:15,555 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:15,555 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:15,555 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:15,555 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:56:15,556 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:56:15,555 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:56:15,555 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:56:15,556 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:56:15,556 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:56:15,556 DEBUG [Thread-4138] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:56:15,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:56:15,556 DEBUG [Thread-4138] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:56:15,556 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:56:15,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:56:15,637 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230416321fe1e1ef574ac38614591a8051cc1f to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230416321fe1e1ef574ac38614591a8051cc1f 2023-04-16 00:56:15,656 DEBUG [Thread-4138] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:56:15,657 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:56:15,658 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:56:15,659 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:56:15,659 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:56:15,659 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:56:15,659 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:56:15,659 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:56:15,659 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:56:15,659 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:15,660 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:15,661 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:15,661 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:56:15,661 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:56:15,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:56:15,661 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:15,661 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:56:15,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:15,661 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:15,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:56:15,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:15,662 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:56:15,662 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:56:15,662 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:56:15,662 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:56:15,662 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:56:15,662 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:56:15,662 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:56:15,662 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:56:15,662 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. 2023-04-16 00:56:15,662 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. started... 2023-04-16 00:56:15,663 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. as already flushing 2023-04-16 00:56:15,663 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. 2023-04-16 00:56:15,663 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:56:15,663 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:56:15,663 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:15,663 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:56:15,663 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:56:15,665 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:15,665 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:15,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:15,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:15,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:15,665 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:15,665 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:56:15,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:15,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:15,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:15,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:15,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:15,667 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:56:15,667 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@63ef4982[Count = 0] remaining members to acquire global barrier 2023-04-16 00:56:15,667 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:56:15,667 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:15,668 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:15,668 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:15,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:15,669 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:15,669 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:56:15,669 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:56:15,669 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:56:15,669 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:56:15,670 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:56:15,670 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:56:15,670 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:56:15,670 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:15,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:15,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:15,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:15,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:15,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:15,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:15,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:15,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:15,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:15,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:15,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:56:15,673 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:56:15,673 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:56:15,673 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:56:15,673 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:56:15,673 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:56:15,674 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:56:15,676 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:15,676 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:56:15,676 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:15,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:15,676 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:15,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:15,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:15,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:15,676 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:56:15,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:56:15,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:15,677 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:15,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:15,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:15,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:15,678 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:15,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:15,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:15,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:15,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:15,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:15,682 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:15,682 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:15,682 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:15,682 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:15,682 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:15,682 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:15,682 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:15,682 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:56:15,682 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:56:15,682 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:56:15,682 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:56:15,682 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:56:15,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:56:15,682 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:56:15,682 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:56:15,682 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:56:15,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:56:15,682 DEBUG [Thread-4138] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:56:15,682 DEBUG [Thread-4138] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:56:15,783 DEBUG [Thread-4138] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:56:15,783 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:56:15,784 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:56:15,785 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:56:15,786 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:56:15,786 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:56:15,786 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:56:15,786 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:56:15,786 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:56:15,786 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:15,786 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:15,787 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:56:15,787 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:15,787 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:56:15,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:56:15,788 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:15,788 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:56:15,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:15,788 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:15,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:56:15,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:15,788 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:56:15,792 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:56:15,792 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:56:15,792 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:56:15,792 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:56:15,792 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:56:15,800 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:56:15,800 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:56:15,800 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. 2023-04-16 00:56:15,800 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. started... 2023-04-16 00:56:15,800 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. as already flushing 2023-04-16 00:56:15,800 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. 2023-04-16 00:56:15,800 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:56:15,800 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:56:15,800 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:15,800 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:56:15,800 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:56:15,801 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:15,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:15,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:15,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:15,802 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:15,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:15,802 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:15,802 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:56:15,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:15,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:15,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:15,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:15,803 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:56:15,803 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1f9e4ff3[Count = 0] remaining members to acquire global barrier 2023-04-16 00:56:15,803 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:56:15,803 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:15,804 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:15,804 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:15,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:15,805 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:56:15,805 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:56:15,805 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:15,805 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:56:15,805 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:56:15,807 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:56:15,807 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:15,807 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:56:15,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:15,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:15,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:15,807 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:56:15,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:15,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:15,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:15,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:15,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:15,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:15,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:15,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:56:15,810 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:56:15,810 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:56:15,810 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:56:15,810 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:56:15,810 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:56:15,810 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:56:15,811 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:15,811 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:15,811 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:56:15,811 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:15,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:15,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:15,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:15,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:15,812 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:56:15,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:56:15,812 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:15,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:15,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:15,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:15,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:15,813 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:15,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:15,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:15,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:15,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:15,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:15,815 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:15,815 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:56:15,815 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:15,815 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:56:15,815 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:56:15,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:56:15,815 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:56:15,815 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:56:15,815 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:56:15,815 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:15,815 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:56:15,816 DEBUG [Thread-4138] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:56:15,816 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:56:15,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:56:15,815 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:15,816 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:15,816 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:15,816 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:15,816 DEBUG [Thread-4138] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:56:15,844 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:15,844 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44562 deadline: 1681606635843, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:15,846 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:15,846 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:15,846 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44564 deadline: 1681606635843, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:15,846 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44556 deadline: 1681606635844, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:15,846 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:15,847 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44554 deadline: 1681606635845, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:15,847 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:15,847 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44548 deadline: 1681606635846, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:15,916 DEBUG [Thread-4138] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:56:15,917 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:56:15,918 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:56:15,919 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:56:15,919 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:56:15,919 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:56:15,919 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:56:15,919 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:56:15,919 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:56:15,919 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:15,919 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:15,921 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:56:15,921 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:15,921 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:56:15,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:56:15,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:15,921 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:15,921 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:56:15,921 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:15,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:56:15,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:15,922 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:56:15,922 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:56:15,922 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:56:15,922 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:56:15,922 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:56:15,922 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:56:15,922 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:56:15,922 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:56:15,923 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. 2023-04-16 00:56:15,923 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. started... 2023-04-16 00:56:15,923 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. as already flushing 2023-04-16 00:56:15,923 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. 2023-04-16 00:56:15,923 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:56:15,923 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:56:15,923 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:15,923 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:56:15,923 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:56:15,924 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:15,924 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:15,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:15,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:15,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:15,925 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:15,925 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:56:15,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:15,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:15,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:15,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:15,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:15,926 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:56:15,926 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5b0100a[Count = 0] remaining members to acquire global barrier 2023-04-16 00:56:15,926 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:56:15,926 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:15,927 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:15,927 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:15,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:15,927 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:56:15,927 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:56:15,927 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:56:15,927 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:15,927 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:56:15,929 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:56:15,929 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:56:15,929 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:15,929 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:56:15,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:15,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:15,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:15,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:15,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:15,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:15,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:15,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:15,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:15,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:15,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:56:15,931 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:56:15,931 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:56:15,931 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:56:15,931 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:56:15,931 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:56:15,931 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:56:15,933 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:15,933 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:15,933 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:56:15,933 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:15,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:15,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:15,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:15,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:15,933 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:56:15,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:15,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:56:15,933 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:15,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:15,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:15,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:15,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:15,935 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:15,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:15,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:15,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:15,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:15,937 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:15,937 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:15,937 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:15,937 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:15,937 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:56:15,937 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:56:15,937 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:56:15,937 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:56:15,937 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:56:15,937 DEBUG [Thread-4138] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:56:15,937 DEBUG [Thread-4138] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:56:15,937 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:15,937 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:15,937 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:56:15,937 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:56:15,937 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:15,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:56:15,938 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:56:15,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:56:16,037 DEBUG [Thread-4138] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:56:16,038 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:56:16,040 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:56:16,041 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:56:16,041 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:56:16,041 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:56:16,041 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:56:16,041 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:56:16,041 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:56:16,041 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:16,041 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:16,042 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=64, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp/A/381fa88c7f654cbf9ad13e96d34818b1 2023-04-16 00:56:16,043 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:56:16,043 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:56:16,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:56:16,043 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:16,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:16,044 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:16,044 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:16,044 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:56:16,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:56:16,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:16,044 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:56:16,045 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:56:16,045 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:56:16,045 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:56:16,045 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:56:16,045 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:56:16,045 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:56:16,045 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:56:16,045 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. 2023-04-16 00:56:16,045 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. started... 2023-04-16 00:56:16,045 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. as already flushing 2023-04-16 00:56:16,045 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. 2023-04-16 00:56:16,045 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:56:16,046 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:56:16,046 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:16,046 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:56:16,046 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:56:16,054 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:16,054 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:16,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:16,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:16,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:16,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:16,054 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=64 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp/B/a148cb76b0584b23afed65b3b63fd481 2023-04-16 00:56:16,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:16,055 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:16,055 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:56:16,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:16,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:16,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:16,056 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:56:16,057 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1720a90d[Count = 0] remaining members to acquire global barrier 2023-04-16 00:56:16,057 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:56:16,057 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:16,058 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:16,058 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:56:16,058 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:16,058 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:16,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:16,059 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:56:16,059 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:56:16,059 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:56:16,060 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:56:16,061 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:56:16,061 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:56:16,061 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:16,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:16,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:16,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:16,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:16,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:16,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:16,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:16,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:16,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:16,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:16,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:56:16,064 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:56:16,064 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:56:16,065 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:56:16,065 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:56:16,065 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:56:16,065 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:56:16,066 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=64 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp/C/17562f43b6914cfca9e885928bcf4d9f 2023-04-16 00:56:16,066 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:16,067 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:56:16,067 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:16,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:16,067 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:16,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:16,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:16,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:16,067 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:56:16,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:56:16,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:16,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:16,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:16,068 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:16,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:16,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:16,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:16,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:16,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:16,070 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:16,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:16,071 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp/A/381fa88c7f654cbf9ad13e96d34818b1 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A/381fa88c7f654cbf9ad13e96d34818b1 2023-04-16 00:56:16,074 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:56:16,074 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:16,074 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A/381fa88c7f654cbf9ad13e96d34818b1, entries=150, sequenceid=64, filesize=25.1 K 2023-04-16 00:56:16,074 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:56:16,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:56:16,074 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:16,074 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:56:16,074 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:56:16,075 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:56:16,075 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:56:16,074 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:16,075 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:56:16,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:56:16,075 DEBUG [Thread-4138] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:56:16,075 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:16,075 DEBUG [Thread-4138] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:56:16,075 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp/B/a148cb76b0584b23afed65b3b63fd481 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/a148cb76b0584b23afed65b3b63fd481 2023-04-16 00:56:16,075 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:16,075 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:16,075 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:16,078 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/a148cb76b0584b23afed65b3b63fd481, entries=150, sequenceid=64, filesize=11.6 K 2023-04-16 00:56:16,079 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp/C/17562f43b6914cfca9e885928bcf4d9f as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/17562f43b6914cfca9e885928bcf4d9f 2023-04-16 00:56:16,082 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/17562f43b6914cfca9e885928bcf4d9f, entries=150, sequenceid=64, filesize=11.6 K 2023-04-16 00:56:16,082 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for d538367277459b70f4a06235dfaacfec in 866ms, sequenceid=64, compaction requested=true 2023-04-16 00:56:16,082 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for d538367277459b70f4a06235dfaacfec: 2023-04-16 00:56:16,082 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 00:56:16,082 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 00:56:16,082 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-16 00:56:16,082 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-16 00:56:16,082 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 00:56:16,085 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 84178 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-16 00:56:16,087 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-16 00:56:16,088 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HStore(1912): d538367277459b70f4a06235dfaacfec/A is initiating minor compaction (all files) 2023-04-16 00:56:16,088 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HStore(1912): d538367277459b70f4a06235dfaacfec/B is initiating minor compaction (all files) 2023-04-16 00:56:16,088 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HRegion(2259): Starting compaction of d538367277459b70f4a06235dfaacfec/B in TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. 2023-04-16 00:56:16,088 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of d538367277459b70f4a06235dfaacfec/A in TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. 2023-04-16 00:56:16,088 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/df0e86e715234f7aa1b3cb278f41ed4c, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/621929dbbab54631aa24fa3dfc6566b4, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/a148cb76b0584b23afed65b3b63fd481] into tmpdir=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp, totalSize=34.8 K 2023-04-16 00:56:16,088 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A/abd5cdebfeef4d6e8e910c771a0c80aa, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A/9f993bd920d148539f837ee17dae4620, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A/381fa88c7f654cbf9ad13e96d34818b1] into tmpdir=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp, totalSize=82.2 K 2023-04-16 00:56:16,089 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting df0e86e715234f7aa1b3cb278f41ed4c, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=16, earliestPutTs=1681606574551 2023-04-16 00:56:16,089 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.Compactor(207): Compacting abd5cdebfeef4d6e8e910c771a0c80aa, keycount=200, bloomtype=ROW, size=31.9 K, encoding=NONE, compression=NONE, seqNum=16, earliestPutTs=1681606574549 2023-04-16 00:56:16,090 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.Compactor(207): Compacting 9f993bd920d148539f837ee17dae4620, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=52, earliestPutTs=1681606574587 2023-04-16 00:56:16,090 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting 621929dbbab54631aa24fa3dfc6566b4, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=52, earliestPutTs=1681606574587 2023-04-16 00:56:16,091 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.Compactor(207): Compacting 381fa88c7f654cbf9ad13e96d34818b1, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=64, earliestPutTs=1681606575215 2023-04-16 00:56:16,091 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting a148cb76b0584b23afed65b3b63fd481, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=64, earliestPutTs=1681606575215 2023-04-16 00:56:16,098 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] throttle.PressureAwareThroughputController(145): d538367277459b70f4a06235dfaacfec#B#compaction#565 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-16 00:56:16,100 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] throttle.PressureAwareThroughputController(145): d538367277459b70f4a06235dfaacfec#A#compaction#564 average throughput is 4.93 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-16 00:56:16,119 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp/B/d0f033214214487fa0a5f40833b417f6 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/d0f033214214487fa0a5f40833b417f6 2023-04-16 00:56:16,122 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in d538367277459b70f4a06235dfaacfec/B of d538367277459b70f4a06235dfaacfec into d0f033214214487fa0a5f40833b417f6(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 00:56:16,122 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for d538367277459b70f4a06235dfaacfec: 2023-04-16 00:56:16,122 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec., storeName=d538367277459b70f4a06235dfaacfec/B, priority=13, startTime=1681606576082; duration=0sec 2023-04-16 00:56:16,122 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 00:56:16,123 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-16 00:56:16,124 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-16 00:56:16,124 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HStore(1912): d538367277459b70f4a06235dfaacfec/C is initiating minor compaction (all files) 2023-04-16 00:56:16,124 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HRegion(2259): Starting compaction of d538367277459b70f4a06235dfaacfec/C in TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. 2023-04-16 00:56:16,124 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/124b949b8e2c4623b6d3c107c5e15ac6, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/edc54e20ad384b238b805bdb755bbfe4, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/17562f43b6914cfca9e885928bcf4d9f] into tmpdir=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp, totalSize=34.8 K 2023-04-16 00:56:16,124 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting 124b949b8e2c4623b6d3c107c5e15ac6, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=16, earliestPutTs=1681606574551 2023-04-16 00:56:16,125 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting edc54e20ad384b238b805bdb755bbfe4, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=52, earliestPutTs=1681606574587 2023-04-16 00:56:16,125 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting 17562f43b6914cfca9e885928bcf4d9f, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=64, earliestPutTs=1681606575215 2023-04-16 00:56:16,131 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] throttle.PressureAwareThroughputController(145): d538367277459b70f4a06235dfaacfec#C#compaction#566 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-16 00:56:16,139 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp/C/c4a61b44ad3345e48ed0e12d54c2a06f as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/c4a61b44ad3345e48ed0e12d54c2a06f 2023-04-16 00:56:16,144 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in d538367277459b70f4a06235dfaacfec/C of d538367277459b70f4a06235dfaacfec into c4a61b44ad3345e48ed0e12d54c2a06f(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 00:56:16,144 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for d538367277459b70f4a06235dfaacfec: 2023-04-16 00:56:16,144 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec., storeName=d538367277459b70f4a06235dfaacfec/C, priority=13, startTime=1681606576082; duration=0sec 2023-04-16 00:56:16,144 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 00:56:16,175 DEBUG [Thread-4138] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:56:16,175 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:56:16,177 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:56:16,178 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:56:16,178 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:56:16,178 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:56:16,178 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:56:16,178 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:56:16,178 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:56:16,179 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:16,179 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:16,180 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:56:16,180 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:16,180 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:56:16,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:56:16,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:16,188 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:16,188 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:56:16,188 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:16,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:56:16,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:16,189 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:56:16,189 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:56:16,189 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:56:16,189 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:56:16,189 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:56:16,189 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:56:16,189 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:56:16,189 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:56:16,189 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. 2023-04-16 00:56:16,190 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. started... 2023-04-16 00:56:16,190 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing d538367277459b70f4a06235dfaacfec 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-16 00:56:16,190 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d538367277459b70f4a06235dfaacfec, store=A 2023-04-16 00:56:16,190 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:56:16,190 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d538367277459b70f4a06235dfaacfec, store=B 2023-04-16 00:56:16,190 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:56:16,190 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d538367277459b70f4a06235dfaacfec, store=C 2023-04-16 00:56:16,190 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:56:16,230 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304168438fabfca784820a64d5ac8cd3053d3 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304168438fabfca784820a64d5ac8cd3053d3 2023-04-16 00:56:16,237 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=86, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp/A/28c735ea22c94a079f00471488c02e8c 2023-04-16 00:56:16,246 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=86 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp/B/be370d9d99b142a2a9d617304ddc164a 2023-04-16 00:56:16,258 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=86 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp/C/acbdf8b5ad7640feacb35f81653f8da2 2023-04-16 00:56:16,262 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp/A/28c735ea22c94a079f00471488c02e8c as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A/28c735ea22c94a079f00471488c02e8c 2023-04-16 00:56:16,266 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A/28c735ea22c94a079f00471488c02e8c, entries=150, sequenceid=86, filesize=25.1 K 2023-04-16 00:56:16,267 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp/B/be370d9d99b142a2a9d617304ddc164a as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/be370d9d99b142a2a9d617304ddc164a 2023-04-16 00:56:16,272 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/be370d9d99b142a2a9d617304ddc164a, entries=150, sequenceid=86, filesize=11.6 K 2023-04-16 00:56:16,272 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp/C/acbdf8b5ad7640feacb35f81653f8da2 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/acbdf8b5ad7640feacb35f81653f8da2 2023-04-16 00:56:16,281 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/acbdf8b5ad7640feacb35f81653f8da2, entries=150, sequenceid=86, filesize=11.6 K 2023-04-16 00:56:16,282 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=0 B/0 for d538367277459b70f4a06235dfaacfec in 92ms, sequenceid=86, compaction requested=false 2023-04-16 00:56:16,282 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for d538367277459b70f4a06235dfaacfec: 2023-04-16 00:56:16,282 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. 2023-04-16 00:56:16,282 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:56:16,282 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:56:16,282 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:16,282 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:56:16,282 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:56:16,284 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:16,284 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:16,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:16,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:16,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:16,284 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:16,284 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:56:16,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:16,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:16,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:16,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:16,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:16,286 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:56:16,286 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@e9a5132[Count = 0] remaining members to acquire global barrier 2023-04-16 00:56:16,286 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:56:16,286 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:16,287 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:16,287 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:16,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:16,287 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:56:16,287 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:56:16,287 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:56:16,287 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:16,287 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:56:16,289 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:56:16,290 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:16,290 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:56:16,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:16,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:16,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:16,290 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:56:16,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:16,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:16,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:16,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:16,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:16,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:16,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:16,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:56:16,294 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:56:16,294 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:56:16,294 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:56:16,294 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:56:16,294 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:56:16,294 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:56:16,296 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:16,296 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:16,296 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:16,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:16,296 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:56:16,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:16,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:16,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:16,296 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:56:16,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:56:16,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:16,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:16,297 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:16,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:16,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:16,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:16,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:16,298 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:16,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:16,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:16,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:16,300 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:56:16,300 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:56:16,300 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:56:16,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:56:16,300 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:16,300 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:16,300 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:16,300 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:16,300 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:16,300 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:16,301 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:16,301 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:56:16,301 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:56:16,301 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:56:16,301 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:56:16,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:56:16,301 DEBUG [Thread-4138] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:56:16,301 DEBUG [Thread-4138] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:56:16,301 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-16 00:56:16,378 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(9158): Flush requested on d538367277459b70f4a06235dfaacfec 2023-04-16 00:56:16,379 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing d538367277459b70f4a06235dfaacfec 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-16 00:56:16,379 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d538367277459b70f4a06235dfaacfec, store=A 2023-04-16 00:56:16,379 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:56:16,379 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d538367277459b70f4a06235dfaacfec, store=B 2023-04-16 00:56:16,379 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:56:16,379 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d538367277459b70f4a06235dfaacfec, store=C 2023-04-16 00:56:16,379 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:56:16,393 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023041657ec1724939a48b68580b99520655865 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041657ec1724939a48b68580b99520655865 2023-04-16 00:56:16,401 DEBUG [Thread-4138] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:56:16,402 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:56:16,404 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:56:16,408 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:56:16,408 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:56:16,408 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:56:16,408 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:56:16,408 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:56:16,408 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:56:16,409 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:16,409 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:16,410 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:56:16,410 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:56:16,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:56:16,410 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:16,410 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:16,410 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:56:16,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:16,411 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:16,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:56:16,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:16,411 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:56:16,416 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:56:16,416 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:56:16,416 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:56:16,416 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:56:16,416 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:56:16,416 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:56:16,416 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:56:16,416 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=96, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp/A/c09f3418d25c46479eaed2c8c09d2740 2023-04-16 00:56:16,416 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. 2023-04-16 00:56:16,417 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. started... 2023-04-16 00:56:16,417 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. as already flushing 2023-04-16 00:56:16,417 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. 2023-04-16 00:56:16,417 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:56:16,417 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:56:16,417 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:16,417 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:56:16,417 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:56:16,420 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:16,420 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:16,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:16,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:16,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:16,421 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=609, cellSize=27885, totalHeapSize=63576, min timestamp=1681606576391, max timestamp=1681606576419Num uniques 859; 2023-04-16 00:56:16,421 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=609, cellSize=27885, totalHeapSize=63576, min timestamp=1681606576391, max timestamp=1681606576419Num uniques 859; 2023-04-16 00:56:16,422 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=609, cellSize=27885, totalHeapSize=63576, min timestamp=1681606576391, max timestamp=1681606576419Num uniques 859; 2023-04-16 00:56:16,422 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=27885, suffix heap size=117064,new segment heap size=63576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=609 2023-04-16 00:56:16,421 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=27885, suffix heap size=117064,new segment heap size=63576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=609 2023-04-16 00:56:16,422 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:16,422 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:56:16,422 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=27885, suffix heap size=117064,new segment heap size=63576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=609 2023-04-16 00:56:16,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:16,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:16,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:16,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:16,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:16,426 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:56:16,426 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@631666c5[Count = 0] remaining members to acquire global barrier 2023-04-16 00:56:16,426 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:56:16,426 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:16,429 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:16,429 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:16,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:16,429 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:56:16,429 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:56:16,429 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:16,429 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:56:16,429 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:56:16,430 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:56:16,430 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:56:16,430 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:56:16,431 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:16,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:16,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:16,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:16,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:16,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:16,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:16,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:16,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:16,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:16,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:16,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:56:16,434 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:56:16,434 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:56:16,434 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:56:16,434 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:56:16,434 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:56:16,434 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:56:16,435 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:16,436 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:16,436 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:16,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:16,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:16,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:16,436 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:56:16,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:16,436 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:56:16,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:56:16,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:16,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:16,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:16,437 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:16,438 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=96 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp/B/5662ac01a6d24740bbfcef0230bcb736 2023-04-16 00:56:16,437 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:16,438 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44548 deadline: 1681606636434, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:16,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:16,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:16,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:16,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:16,440 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:16,440 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:16,440 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44554 deadline: 1681606636435, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:16,440 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:16,440 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44564 deadline: 1681606636436, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:16,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:16,441 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:16,441 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44562 deadline: 1681606636438, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:16,441 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:16,442 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44556 deadline: 1681606636438, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:16,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:16,444 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:16,445 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:16,445 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:16,445 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:16,445 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:56:16,445 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:56:16,445 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:56:16,445 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:56:16,445 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:56:16,445 DEBUG [Thread-4138] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:56:16,445 DEBUG [Thread-4138] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:56:16,445 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:16,445 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:56:16,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:56:16,445 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:16,446 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:16,446 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:56:16,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:56:16,454 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=96 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp/C/ffe29d2d730248dd8a659ca42454aa97 2023-04-16 00:56:16,466 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp/A/c09f3418d25c46479eaed2c8c09d2740 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A/c09f3418d25c46479eaed2c8c09d2740 2023-04-16 00:56:16,470 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A/c09f3418d25c46479eaed2c8c09d2740, entries=350, sequenceid=96, filesize=52.3 K 2023-04-16 00:56:16,471 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp/B/5662ac01a6d24740bbfcef0230bcb736 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/5662ac01a6d24740bbfcef0230bcb736 2023-04-16 00:56:16,474 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/5662ac01a6d24740bbfcef0230bcb736, entries=100, sequenceid=96, filesize=9.3 K 2023-04-16 00:56:16,475 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp/C/ffe29d2d730248dd8a659ca42454aa97 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/ffe29d2d730248dd8a659ca42454aa97 2023-04-16 00:56:16,478 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/ffe29d2d730248dd8a659ca42454aa97, entries=100, sequenceid=96, filesize=9.3 K 2023-04-16 00:56:16,479 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=154.31 KB/158010 for d538367277459b70f4a06235dfaacfec in 101ms, sequenceid=96, compaction requested=true 2023-04-16 00:56:16,479 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for d538367277459b70f4a06235dfaacfec: 2023-04-16 00:56:16,479 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 00:56:16,480 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 00:56:16,480 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-16 00:56:16,480 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 3 compacting, 2 eligible, 16 blocking 2023-04-16 00:56:16,480 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-04-16 00:56:16,480 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-04-16 00:56:16,480 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. because compaction request was cancelled 2023-04-16 00:56:16,481 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-16 00:56:16,482 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 33417 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-16 00:56:16,482 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HStore(1912): d538367277459b70f4a06235dfaacfec/B is initiating minor compaction (all files) 2023-04-16 00:56:16,483 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HRegion(2259): Starting compaction of d538367277459b70f4a06235dfaacfec/B in TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. 2023-04-16 00:56:16,483 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/d0f033214214487fa0a5f40833b417f6, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/be370d9d99b142a2a9d617304ddc164a, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/5662ac01a6d24740bbfcef0230bcb736] into tmpdir=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp, totalSize=32.6 K 2023-04-16 00:56:16,484 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting d0f033214214487fa0a5f40833b417f6, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=64, earliestPutTs=1681606575215 2023-04-16 00:56:16,484 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting be370d9d99b142a2a9d617304ddc164a, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=86, earliestPutTs=1681606575228 2023-04-16 00:56:16,484 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting 5662ac01a6d24740bbfcef0230bcb736, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=96, earliestPutTs=1681606576376 2023-04-16 00:56:16,502 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] throttle.PressureAwareThroughputController(145): d538367277459b70f4a06235dfaacfec#B#compaction#573 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-16 00:56:16,537 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp/B/697a03eb36f64a8ba6a8436f67894577 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/697a03eb36f64a8ba6a8436f67894577 2023-04-16 00:56:16,542 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in d538367277459b70f4a06235dfaacfec/B of d538367277459b70f4a06235dfaacfec into 697a03eb36f64a8ba6a8436f67894577(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 00:56:16,542 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for d538367277459b70f4a06235dfaacfec: 2023-04-16 00:56:16,542 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec., storeName=d538367277459b70f4a06235dfaacfec/B, priority=13, startTime=1681606576480; duration=0sec 2023-04-16 00:56:16,543 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 00:56:16,543 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-16 00:56:16,543 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(9158): Flush requested on d538367277459b70f4a06235dfaacfec 2023-04-16 00:56:16,544 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 33417 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-16 00:56:16,544 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing d538367277459b70f4a06235dfaacfec 3/3 column families, dataSize=161.02 KB heapSize=422.48 KB 2023-04-16 00:56:16,544 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HStore(1912): d538367277459b70f4a06235dfaacfec/C is initiating minor compaction (all files) 2023-04-16 00:56:16,544 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HRegion(2259): Starting compaction of d538367277459b70f4a06235dfaacfec/C in TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. 2023-04-16 00:56:16,544 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d538367277459b70f4a06235dfaacfec, store=A 2023-04-16 00:56:16,544 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/c4a61b44ad3345e48ed0e12d54c2a06f, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/acbdf8b5ad7640feacb35f81653f8da2, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/ffe29d2d730248dd8a659ca42454aa97] into tmpdir=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp, totalSize=32.6 K 2023-04-16 00:56:16,545 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 00:56:16,545 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d538367277459b70f4a06235dfaacfec, store=B 2023-04-16 00:56:16,545 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 00:56:16,545 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d538367277459b70f4a06235dfaacfec, store=C 2023-04-16 00:56:16,545 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 00:56:16,545 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting c4a61b44ad3345e48ed0e12d54c2a06f, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=64, earliestPutTs=1681606575215 2023-04-16 00:56:16,545 DEBUG [Thread-4138] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:56:16,548 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting acbdf8b5ad7640feacb35f81653f8da2, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=86, earliestPutTs=1681606575228 2023-04-16 00:56:16,548 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:56:16,549 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting ffe29d2d730248dd8a659ca42454aa97, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=96, earliestPutTs=1681606576376 2023-04-16 00:56:16,552 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp/A/3d98fe119a4c4b74818836f84975adfb as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A/3d98fe119a4c4b74818836f84975adfb 2023-04-16 00:56:16,554 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:56:16,554 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:16,554 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44554 deadline: 1681606636550, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:16,554 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:16,555 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44564 deadline: 1681606636551, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:16,555 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:56:16,555 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:56:16,555 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:56:16,555 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:56:16,555 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:56:16,555 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:56:16,556 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:16,556 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:16,558 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:56:16,558 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:56:16,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:56:16,558 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:16,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:16,559 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:16,559 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:56:16,559 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:16,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:56:16,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:16,559 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:56:16,559 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in d538367277459b70f4a06235dfaacfec/A of d538367277459b70f4a06235dfaacfec into 3d98fe119a4c4b74818836f84975adfb(size=25.2 K), total size for store is 102.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 00:56:16,559 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for d538367277459b70f4a06235dfaacfec: 2023-04-16 00:56:16,559 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec., storeName=d538367277459b70f4a06235dfaacfec/A, priority=13, startTime=1681606576082; duration=0sec 2023-04-16 00:56:16,560 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 00:56:16,560 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:16,560 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44562 deadline: 1681606636554, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:16,560 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:16,560 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:16,561 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:56:16,561 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44548 deadline: 1681606636555, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:16,561 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:56:16,561 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:56:16,560 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44556 deadline: 1681606636554, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:16,561 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] throttle.PressureAwareThroughputController(145): d538367277459b70f4a06235dfaacfec#C#compaction#575 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-16 00:56:16,561 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:56:16,561 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:56:16,564 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:56:16,564 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:56:16,564 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. 2023-04-16 00:56:16,564 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. started... 2023-04-16 00:56:16,564 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. as already flushing 2023-04-16 00:56:16,564 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. 2023-04-16 00:56:16,564 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:56:16,564 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:56:16,564 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:16,564 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:56:16,564 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:56:16,566 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:16,566 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:16,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:16,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:16,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:16,566 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:16,566 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:56:16,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:16,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:16,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:16,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:16,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:16,567 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:56:16,567 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@66231aed[Count = 0] remaining members to acquire global barrier 2023-04-16 00:56:16,567 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:56:16,567 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:16,574 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:16,574 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:16,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:16,575 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:56:16,577 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:56:16,577 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:56:16,577 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:16,577 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:56:16,578 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:16,578 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:56:16,578 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:56:16,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:16,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:16,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:16,578 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:56:16,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:16,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:16,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:16,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:16,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:16,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:16,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:16,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:56:16,585 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:56:16,585 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:56:16,585 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:56:16,585 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:56:16,585 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:56:16,585 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:56:16,587 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:16,587 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:16,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:16,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:16,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:16,587 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:16,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:16,587 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:56:16,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:16,588 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:56:16,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:56:16,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:16,589 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:16,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:16,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:16,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:16,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:16,591 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:16,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:16,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:16,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:16,593 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:56:16,593 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:16,593 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:56:16,593 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:56:16,593 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:56:16,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:56:16,593 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:56:16,593 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:16,593 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:56:16,593 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:16,594 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:16,594 DEBUG [Thread-4138] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:56:16,594 DEBUG [Thread-4138] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:56:16,594 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:16,594 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:16,594 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:16,594 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:56:16,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:56:16,658 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:16,658 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44564 deadline: 1681606636656, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:16,658 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:16,658 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44554 deadline: 1681606636656, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:16,664 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:16,664 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44562 deadline: 1681606636662, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:16,664 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:16,664 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44548 deadline: 1681606636662, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:16,664 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:16,665 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44556 deadline: 1681606636662, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:16,694 DEBUG [Thread-4138] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:56:16,694 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:56:16,696 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:56:16,697 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:56:16,697 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:56:16,697 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:56:16,697 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:56:16,697 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:56:16,697 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:56:16,698 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:16,698 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:16,700 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:56:16,700 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:16,700 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:56:16,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:56:16,700 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:16,700 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:56:16,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:16,701 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:16,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:56:16,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:16,701 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:56:16,701 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:56:16,701 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:56:16,701 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:56:16,701 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:56:16,702 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:56:16,702 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:56:16,702 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:56:16,702 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. 2023-04-16 00:56:16,702 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. started... 2023-04-16 00:56:16,702 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. as already flushing 2023-04-16 00:56:16,702 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. 2023-04-16 00:56:16,702 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:56:16,702 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:56:16,702 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:16,702 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:56:16,702 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:56:16,703 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:16,703 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:16,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:16,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:16,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:16,704 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:16,704 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:56:16,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:16,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:16,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:16,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:16,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:16,705 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:56:16,705 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7459bfe5[Count = 0] remaining members to acquire global barrier 2023-04-16 00:56:16,705 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:56:16,705 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:16,706 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:16,706 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:16,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:16,706 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:56:16,706 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:56:16,706 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:56:16,707 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:16,707 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:56:16,708 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:16,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:16,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:16,708 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:56:16,708 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:56:16,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:16,708 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:56:16,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:16,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:16,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:16,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:16,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:16,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:16,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:16,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:56:16,710 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:56:16,710 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:56:16,710 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:56:16,710 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:56:16,710 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:56:16,710 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:56:16,713 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:16,713 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:16,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:16,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:16,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:16,713 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:16,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:16,713 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:56:16,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:16,714 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:56:16,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:56:16,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:16,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:16,714 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:16,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:16,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:16,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:16,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:16,715 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:16,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:16,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:16,717 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:16,717 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:16,717 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:56:16,717 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:56:16,718 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:56:16,717 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:56:16,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:56:16,717 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:56:16,717 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:56:16,717 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:16,718 DEBUG [Thread-4138] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:56:16,718 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:56:16,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:56:16,718 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:16,718 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:16,718 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:16,718 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:16,718 DEBUG [Thread-4138] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:56:16,818 DEBUG [Thread-4138] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:56:16,819 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:56:16,820 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:56:16,821 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:56:16,822 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:56:16,822 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:56:16,822 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:56:16,822 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:56:16,822 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:56:16,822 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:16,822 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:16,823 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:16,823 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:56:16,824 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:56:16,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:56:16,824 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:16,824 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:56:16,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:16,824 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:16,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:56:16,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:16,825 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:56:16,825 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:56:16,825 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:56:16,825 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:56:16,825 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:56:16,825 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:56:16,825 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:56:16,825 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. 2023-04-16 00:56:16,825 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. started... 2023-04-16 00:56:16,825 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. as already flushing 2023-04-16 00:56:16,825 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. 2023-04-16 00:56:16,825 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:56:16,825 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:56:16,825 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:16,825 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:56:16,825 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:56:16,827 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:16,827 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:16,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:16,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:16,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:16,827 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:16,827 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:56:16,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:16,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:16,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:16,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:16,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:16,829 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:56:16,829 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2896639d[Count = 0] remaining members to acquire global barrier 2023-04-16 00:56:16,829 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:56:16,829 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:16,830 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:16,830 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:16,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:16,831 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:56:16,831 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:56:16,831 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:56:16,831 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:16,831 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:56:16,832 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:16,832 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:56:16,832 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:56:16,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:16,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:16,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:16,832 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:56:16,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:16,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:16,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:16,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:16,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:16,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:16,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:16,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:56:16,834 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:56:16,834 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:56:16,834 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:56:16,835 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:56:16,835 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:56:16,835 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:56:16,836 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:16,836 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:16,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:16,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:16,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:16,836 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:16,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:16,836 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:56:16,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:16,837 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:56:16,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:56:16,837 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:16,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:16,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:16,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:16,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:16,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:16,838 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:16,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:16,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:16,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:16,840 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:56:16,840 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:56:16,840 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:56:16,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:56:16,841 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:16,841 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:16,841 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:16,841 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:56:16,841 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:56:16,841 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:56:16,841 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:56:16,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:56:16,841 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:16,841 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:16,841 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:16,841 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:16,841 DEBUG [Thread-4138] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:56:16,841 DEBUG [Thread-4138] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:56:16,862 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:16,862 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:16,862 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44554 deadline: 1681606636859, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:16,862 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44564 deadline: 1681606636859, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:16,866 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:16,867 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44562 deadline: 1681606636865, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:16,867 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:16,867 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44548 deadline: 1681606636865, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:16,868 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:16,868 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44556 deadline: 1681606636866, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:16,941 DEBUG [Thread-4138] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:56:16,942 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:56:16,943 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:56:16,944 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:56:16,944 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:56:16,944 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:56:16,944 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:56:16,945 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:56:16,945 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:56:16,945 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:16,945 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:16,947 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:16,947 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:56:16,948 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:56:16,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:56:16,948 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:16,948 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:56:16,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:16,948 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:16,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:56:16,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:16,949 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:56:16,949 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:56:16,949 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:56:16,949 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:56:16,949 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:56:16,949 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:56:16,949 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:56:16,949 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:56:16,950 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. 2023-04-16 00:56:16,950 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. started... 2023-04-16 00:56:16,950 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. as already flushing 2023-04-16 00:56:16,950 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. 2023-04-16 00:56:16,950 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:56:16,950 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:56:16,950 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:16,950 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:56:16,950 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:56:16,951 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:16,951 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:16,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:16,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:16,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:16,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:16,952 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:16,952 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:56:16,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:16,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:16,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:16,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:16,953 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:56:16,953 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@78661804[Count = 0] remaining members to acquire global barrier 2023-04-16 00:56:16,953 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:56:16,953 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:16,954 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:16,954 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:16,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:16,954 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:56:16,954 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:56:16,954 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:16,954 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:56:16,954 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:56:16,962 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:56:16,962 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:16,962 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:56:16,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:16,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:16,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:16,962 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:56:16,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:16,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:16,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:16,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:16,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:16,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:16,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:16,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:56:16,965 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:56:16,965 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:56:16,965 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:56:16,965 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:56:16,965 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:56:16,965 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:56:16,967 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:16,967 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:16,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:16,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:16,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:16,967 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:16,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:16,967 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:56:16,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:16,967 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:56:16,967 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:16,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:56:16,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:16,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:16,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:16,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:16,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:16,969 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:16,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:16,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:16,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:16,971 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:56:16,971 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:16,971 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:56:16,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:56:16,971 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:16,971 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:56:16,971 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:16,971 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:16,971 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:16,971 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:16,971 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:56:16,971 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:56:16,971 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:56:16,971 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:16,971 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:56:16,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:56:16,972 DEBUG [Thread-4138] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:56:16,972 DEBUG [Thread-4138] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:56:16,978 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023041626c00b2bb38249acba7a88cba3516a19 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041626c00b2bb38249acba7a88cba3516a19 2023-04-16 00:56:16,983 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=129, memsize=53.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp/A/cb9acde7623542dc9f35614296829d70 2023-04-16 00:56:16,984 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp/C/c0dac761019a47d4a1a0ab2d45f5f87d as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/c0dac761019a47d4a1a0ab2d45f5f87d 2023-04-16 00:56:16,991 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in d538367277459b70f4a06235dfaacfec/C of d538367277459b70f4a06235dfaacfec into c0dac761019a47d4a1a0ab2d45f5f87d(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 00:56:16,991 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for d538367277459b70f4a06235dfaacfec: 2023-04-16 00:56:16,991 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec., storeName=d538367277459b70f4a06235dfaacfec/C, priority=13, startTime=1681606576480; duration=0sec 2023-04-16 00:56:16,991 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 00:56:16,993 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=53.67 KB at sequenceid=129 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp/B/b931b8c998cf402d837c49a53305f3ad 2023-04-16 00:56:17,003 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=53.67 KB at sequenceid=129 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp/C/853941a538f34158835a61f3098bdaba 2023-04-16 00:56:17,006 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp/A/cb9acde7623542dc9f35614296829d70 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A/cb9acde7623542dc9f35614296829d70 2023-04-16 00:56:17,010 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A/cb9acde7623542dc9f35614296829d70, entries=200, sequenceid=129, filesize=32.0 K 2023-04-16 00:56:17,011 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp/B/b931b8c998cf402d837c49a53305f3ad as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/b931b8c998cf402d837c49a53305f3ad 2023-04-16 00:56:17,014 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/b931b8c998cf402d837c49a53305f3ad, entries=150, sequenceid=129, filesize=11.7 K 2023-04-16 00:56:17,015 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp/C/853941a538f34158835a61f3098bdaba as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/853941a538f34158835a61f3098bdaba 2023-04-16 00:56:17,017 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/853941a538f34158835a61f3098bdaba, entries=150, sequenceid=129, filesize=11.7 K 2023-04-16 00:56:17,018 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~161.02 KB/164880, heapSize ~422.44 KB/432576, currentSize=33.54 KB/34350 for d538367277459b70f4a06235dfaacfec in 474ms, sequenceid=129, compaction requested=true 2023-04-16 00:56:17,018 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for d538367277459b70f4a06235dfaacfec: 2023-04-16 00:56:17,018 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 00:56:17,020 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-16 00:56:17,021 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 2 store files, 0 compacting, 2 eligible, 16 blocking 2023-04-16 00:56:17,021 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 00:56:17,022 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 00:56:17,023 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-04-16 00:56:17,023 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-04-16 00:56:17,023 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. because compaction request was cancelled 2023-04-16 00:56:17,023 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 2 store files, 0 compacting, 2 eligible, 16 blocking 2023-04-16 00:56:17,024 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 137837 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-16 00:56:17,024 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HStore(1912): d538367277459b70f4a06235dfaacfec/A is initiating minor compaction (all files) 2023-04-16 00:56:17,024 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HRegion(2259): Starting compaction of d538367277459b70f4a06235dfaacfec/A in TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. 2023-04-16 00:56:17,024 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A/3d98fe119a4c4b74818836f84975adfb, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A/28c735ea22c94a079f00471488c02e8c, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A/c09f3418d25c46479eaed2c8c09d2740, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A/cb9acde7623542dc9f35614296829d70] into tmpdir=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp, totalSize=134.6 K 2023-04-16 00:56:17,024 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-04-16 00:56:17,025 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-04-16 00:56:17,025 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. because compaction request was cancelled 2023-04-16 00:56:17,025 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting 3d98fe119a4c4b74818836f84975adfb, keycount=150, bloomtype=ROW, size=25.2 K, encoding=NONE, compression=NONE, seqNum=64, earliestPutTs=1681606575215 2023-04-16 00:56:17,025 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting 28c735ea22c94a079f00471488c02e8c, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=86, earliestPutTs=1681606575228 2023-04-16 00:56:17,026 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting c09f3418d25c46479eaed2c8c09d2740, keycount=350, bloomtype=ROW, size=52.3 K, encoding=NONE, compression=NONE, seqNum=96, earliestPutTs=1681606576367 2023-04-16 00:56:17,026 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting cb9acde7623542dc9f35614296829d70, keycount=200, bloomtype=ROW, size=32.0 K, encoding=NONE, compression=NONE, seqNum=129, earliestPutTs=1681606576433 2023-04-16 00:56:17,037 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] throttle.PressureAwareThroughputController(145): d538367277459b70f4a06235dfaacfec#A#compaction#578 average throughput is 6.57 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-16 00:56:17,072 DEBUG [Thread-4138] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:56:17,072 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:56:17,074 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:56:17,075 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:56:17,075 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:56:17,075 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:56:17,075 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:56:17,076 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:56:17,076 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:56:17,076 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:17,076 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:17,077 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:56:17,077 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:56:17,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:56:17,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:17,078 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:17,078 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:17,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:56:17,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:17,078 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:56:17,078 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:56:17,078 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:56:17,078 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:56:17,078 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:17,079 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:56:17,079 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:56:17,079 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:56:17,079 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:56:17,079 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:56:17,079 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. 2023-04-16 00:56:17,079 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. started... 2023-04-16 00:56:17,079 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing d538367277459b70f4a06235dfaacfec 3/3 column families, dataSize=33.54 KB heapSize=100.36 KB 2023-04-16 00:56:17,079 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d538367277459b70f4a06235dfaacfec, store=A 2023-04-16 00:56:17,079 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:56:17,079 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d538367277459b70f4a06235dfaacfec, store=B 2023-04-16 00:56:17,080 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:56:17,080 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d538367277459b70f4a06235dfaacfec, store=C 2023-04-16 00:56:17,080 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:56:17,091 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230416c10c8b34f3644180ba114a0b6a098a55 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230416c10c8b34f3644180ba114a0b6a098a55 2023-04-16 00:56:17,110 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=139, memsize=11.2 K, hasBloomFilter=true, into tmp file hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp/A/25122e76a3ac47b8accde73fc4baa0c1 2023-04-16 00:56:17,126 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=139 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp/B/7b6da3ebcd154ab7af07a6697dd4a150 2023-04-16 00:56:17,150 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=139 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp/C/8d8c35a7c8bf42e4862f34c5e071664b 2023-04-16 00:56:17,154 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp/A/25122e76a3ac47b8accde73fc4baa0c1 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A/25122e76a3ac47b8accde73fc4baa0c1 2023-04-16 00:56:17,159 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A/25122e76a3ac47b8accde73fc4baa0c1, entries=150, sequenceid=139, filesize=25.3 K 2023-04-16 00:56:17,160 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp/B/7b6da3ebcd154ab7af07a6697dd4a150 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/7b6da3ebcd154ab7af07a6697dd4a150 2023-04-16 00:56:17,168 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/7b6da3ebcd154ab7af07a6697dd4a150, entries=150, sequenceid=139, filesize=11.8 K 2023-04-16 00:56:17,168 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp/C/8d8c35a7c8bf42e4862f34c5e071664b as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/8d8c35a7c8bf42e4862f34c5e071664b 2023-04-16 00:56:17,173 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/8d8c35a7c8bf42e4862f34c5e071664b, entries=150, sequenceid=139, filesize=11.8 K 2023-04-16 00:56:17,174 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~100.31 KB/102720, currentSize=0 B/0 for d538367277459b70f4a06235dfaacfec in 95ms, sequenceid=139, compaction requested=true 2023-04-16 00:56:17,174 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for d538367277459b70f4a06235dfaacfec: 2023-04-16 00:56:17,174 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. 2023-04-16 00:56:17,174 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:56:17,174 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:56:17,174 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:17,176 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:56:17,176 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:56:17,178 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:17,178 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:17,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:17,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:17,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:17,178 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:17,178 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:56:17,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:17,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:17,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:17,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:17,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:17,179 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:56:17,179 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6ec9197[Count = 0] remaining members to acquire global barrier 2023-04-16 00:56:17,179 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:56:17,180 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:17,190 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:17,190 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:17,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:17,191 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:56:17,191 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:56:17,191 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:56:17,191 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:17,191 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:56:17,192 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:17,192 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:56:17,192 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:56:17,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:17,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:17,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:17,192 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:56:17,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:17,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:17,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:17,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:17,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:17,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:17,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:17,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:56:17,195 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:56:17,195 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:56:17,195 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:56:17,195 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:56:17,195 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:56:17,196 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:56:17,197 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:17,197 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:17,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:17,197 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:17,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:17,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:17,197 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:56:17,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:17,198 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:17,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:17,198 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:56:17,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:56:17,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:17,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:17,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:17,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:17,199 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:17,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:17,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:17,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:17,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:17,203 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:17,203 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:17,203 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:17,203 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:17,204 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:17,204 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:17,204 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:56:17,203 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:56:17,204 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:56:17,204 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:17,204 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:56:17,204 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:56:17,204 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:56:17,204 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:56:17,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:56:17,204 DEBUG [Thread-4138] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:56:17,204 DEBUG [Thread-4138] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:56:17,204 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:56:17,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:56:17,223 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(9158): Flush requested on d538367277459b70f4a06235dfaacfec 2023-04-16 00:56:17,223 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing d538367277459b70f4a06235dfaacfec 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-16 00:56:17,224 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d538367277459b70f4a06235dfaacfec, store=A 2023-04-16 00:56:17,224 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:56:17,224 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d538367277459b70f4a06235dfaacfec, store=B 2023-04-16 00:56:17,224 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:56:17,224 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d538367277459b70f4a06235dfaacfec, store=C 2023-04-16 00:56:17,224 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:56:17,240 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304163c5b5fdb2a96404ab76a3e861a49bed1 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304163c5b5fdb2a96404ab76a3e861a49bed1 2023-04-16 00:56:17,247 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:17,247 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44564 deadline: 1681606637243, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:17,248 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:17,248 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44548 deadline: 1681606637244, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:17,248 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681606577242, max timestamp=1681606577246Num uniques 859; 2023-04-16 00:56:17,248 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:17,248 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44562 deadline: 1681606637245, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:17,249 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681606577242, max timestamp=1681606577246Num uniques 859; 2023-04-16 00:56:17,249 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-16 00:56:17,248 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-16 00:56:17,248 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681606577242, max timestamp=1681606577246Num uniques 859; 2023-04-16 00:56:17,249 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-16 00:56:17,252 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=150, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp/A/1d92c3f646be4e918c53aa9745ef7bb5 2023-04-16 00:56:17,274 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=150 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp/B/4e5fb4ddba9c48bd9313d6d4be4dcd3c 2023-04-16 00:56:17,277 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:17,277 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44556 deadline: 1681606637273, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:17,284 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:17,284 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44554 deadline: 1681606637280, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:17,289 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=150 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp/C/61233cb6892543d7a2bf50475066ea37 2023-04-16 00:56:17,292 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp/A/1d92c3f646be4e918c53aa9745ef7bb5 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A/1d92c3f646be4e918c53aa9745ef7bb5 2023-04-16 00:56:17,296 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A/1d92c3f646be4e918c53aa9745ef7bb5, entries=400, sequenceid=150, filesize=59.4 K 2023-04-16 00:56:17,296 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp/B/4e5fb4ddba9c48bd9313d6d4be4dcd3c as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/4e5fb4ddba9c48bd9313d6d4be4dcd3c 2023-04-16 00:56:17,299 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/4e5fb4ddba9c48bd9313d6d4be4dcd3c, entries=150, sequenceid=150, filesize=11.8 K 2023-04-16 00:56:17,300 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp/C/61233cb6892543d7a2bf50475066ea37 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/61233cb6892543d7a2bf50475066ea37 2023-04-16 00:56:17,303 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/61233cb6892543d7a2bf50475066ea37, entries=150, sequenceid=150, filesize=11.8 K 2023-04-16 00:56:17,303 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for d538367277459b70f4a06235dfaacfec in 80ms, sequenceid=150, compaction requested=true 2023-04-16 00:56:17,303 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for d538367277459b70f4a06235dfaacfec: 2023-04-16 00:56:17,303 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 00:56:17,303 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 00:56:17,304 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-16 00:56:17,304 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 4 compacting, 2 eligible, 16 blocking 2023-04-16 00:56:17,304 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-04-16 00:56:17,304 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-04-16 00:56:17,304 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. because compaction request was cancelled 2023-04-16 00:56:17,304 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-16 00:56:17,304 DEBUG [Thread-4138] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:56:17,305 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:56:17,305 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48100 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-16 00:56:17,306 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HStore(1912): d538367277459b70f4a06235dfaacfec/B is initiating minor compaction (all files) 2023-04-16 00:56:17,306 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of d538367277459b70f4a06235dfaacfec/B in TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. 2023-04-16 00:56:17,306 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/697a03eb36f64a8ba6a8436f67894577, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/b931b8c998cf402d837c49a53305f3ad, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/7b6da3ebcd154ab7af07a6697dd4a150, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/4e5fb4ddba9c48bd9313d6d4be4dcd3c] into tmpdir=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp, totalSize=47.0 K 2023-04-16 00:56:17,306 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.Compactor(207): Compacting 697a03eb36f64a8ba6a8436f67894577, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=96, earliestPutTs=1681606575230 2023-04-16 00:56:17,306 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.Compactor(207): Compacting b931b8c998cf402d837c49a53305f3ad, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=129, earliestPutTs=1681606576433 2023-04-16 00:56:17,307 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.Compactor(207): Compacting 7b6da3ebcd154ab7af07a6697dd4a150, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=139, earliestPutTs=1681606576552 2023-04-16 00:56:17,307 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.Compactor(207): Compacting 4e5fb4ddba9c48bd9313d6d4be4dcd3c, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=150, earliestPutTs=1681606577221 2023-04-16 00:56:17,310 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:56:17,311 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:56:17,311 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:56:17,311 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:56:17,311 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:56:17,311 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:56:17,311 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:56:17,312 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:17,312 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:17,313 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:56:17,313 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:56:17,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:56:17,313 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:17,314 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:17,314 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:56:17,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:17,314 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:17,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:56:17,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:17,315 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:56:17,321 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:56:17,321 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:56:17,321 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:56:17,321 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:56:17,321 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:56:17,323 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] throttle.PressureAwareThroughputController(145): d538367277459b70f4a06235dfaacfec#B#compaction#585 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-16 00:56:17,328 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:56:17,328 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:56:17,328 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. 2023-04-16 00:56:17,328 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. started... 2023-04-16 00:56:17,328 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing d538367277459b70f4a06235dfaacfec 3/3 column families, dataSize=127.47 KB heapSize=360.38 KB 2023-04-16 00:56:17,328 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d538367277459b70f4a06235dfaacfec, store=A 2023-04-16 00:56:17,328 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 00:56:17,328 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d538367277459b70f4a06235dfaacfec, store=B 2023-04-16 00:56:17,329 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 00:56:17,329 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d538367277459b70f4a06235dfaacfec, store=C 2023-04-16 00:56:17,329 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 00:56:17,353 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(9158): Flush requested on d538367277459b70f4a06235dfaacfec 2023-04-16 00:56:17,353 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. as already flushing 2023-04-16 00:56:17,366 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230416590e5ea7e5674fcd96e8a4f0eff6b272 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230416590e5ea7e5674fcd96e8a4f0eff6b272 2023-04-16 00:56:17,397 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:17,398 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44556 deadline: 1681606637378, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:17,398 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:17,398 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44554 deadline: 1681606637385, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:17,398 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:17,398 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44548 deadline: 1681606637392, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:17,398 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:17,398 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44564 deadline: 1681606637392, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:17,398 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:17,399 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44562 deadline: 1681606637392, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:17,458 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp/A/a43dd4d44481416c85d473f9b3e6639f as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A/a43dd4d44481416c85d473f9b3e6639f 2023-04-16 00:56:17,463 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in d538367277459b70f4a06235dfaacfec/A of d538367277459b70f4a06235dfaacfec into a43dd4d44481416c85d473f9b3e6639f(size=25.4 K), total size for store is 110.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 00:56:17,463 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for d538367277459b70f4a06235dfaacfec: 2023-04-16 00:56:17,463 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec., storeName=d538367277459b70f4a06235dfaacfec/A, priority=12, startTime=1681606577018; duration=0sec 2023-04-16 00:56:17,463 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 00:56:17,463 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-16 00:56:17,464 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48100 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-16 00:56:17,464 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HStore(1912): d538367277459b70f4a06235dfaacfec/C is initiating minor compaction (all files) 2023-04-16 00:56:17,464 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HRegion(2259): Starting compaction of d538367277459b70f4a06235dfaacfec/C in TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. 2023-04-16 00:56:17,464 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/c0dac761019a47d4a1a0ab2d45f5f87d, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/853941a538f34158835a61f3098bdaba, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/8d8c35a7c8bf42e4862f34c5e071664b, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/61233cb6892543d7a2bf50475066ea37] into tmpdir=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp, totalSize=47.0 K 2023-04-16 00:56:17,465 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting c0dac761019a47d4a1a0ab2d45f5f87d, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=96, earliestPutTs=1681606575230 2023-04-16 00:56:17,465 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting 853941a538f34158835a61f3098bdaba, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=129, earliestPutTs=1681606576433 2023-04-16 00:56:17,465 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting 8d8c35a7c8bf42e4862f34c5e071664b, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=139, earliestPutTs=1681606576552 2023-04-16 00:56:17,466 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting 61233cb6892543d7a2bf50475066ea37, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=150, earliestPutTs=1681606577221 2023-04-16 00:56:17,472 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] throttle.PressureAwareThroughputController(145): d538367277459b70f4a06235dfaacfec#C#compaction#587 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-16 00:56:17,483 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp/C/259d737a359841719c910c2f0e8fc933 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/259d737a359841719c910c2f0e8fc933 2023-04-16 00:56:17,488 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in d538367277459b70f4a06235dfaacfec/C of d538367277459b70f4a06235dfaacfec into 259d737a359841719c910c2f0e8fc933(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 00:56:17,488 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for d538367277459b70f4a06235dfaacfec: 2023-04-16 00:56:17,488 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec., storeName=d538367277459b70f4a06235dfaacfec/C, priority=12, startTime=1681606577304; duration=0sec 2023-04-16 00:56:17,488 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 00:56:17,501 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:17,501 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44548 deadline: 1681606637499, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:17,503 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:17,503 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44564 deadline: 1681606637499, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:17,503 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:17,503 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44562 deadline: 1681606637499, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:17,603 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:17,603 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:17,603 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44556 deadline: 1681606637600, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:17,603 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44554 deadline: 1681606637600, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:17,705 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:17,705 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44548 deadline: 1681606637703, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:17,707 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:17,707 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44564 deadline: 1681606637705, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:17,710 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:17,710 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44562 deadline: 1681606637708, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:17,745 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp/B/ee3926f1cb034130a58d1315408ab2f1 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/ee3926f1cb034130a58d1315408ab2f1 2023-04-16 00:56:17,749 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in d538367277459b70f4a06235dfaacfec/B of d538367277459b70f4a06235dfaacfec into ee3926f1cb034130a58d1315408ab2f1(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 00:56:17,749 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for d538367277459b70f4a06235dfaacfec: 2023-04-16 00:56:17,749 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec., storeName=d538367277459b70f4a06235dfaacfec/B, priority=12, startTime=1681606577303; duration=0sec 2023-04-16 00:56:17,749 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 00:56:17,773 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=185, memsize=42.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp/A/21025b18a78649a88ca998e1d8ebae26 2023-04-16 00:56:17,909 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:17,909 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44554 deadline: 1681606637905, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:17,909 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:17,909 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44556 deadline: 1681606637906, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:18,011 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:18,011 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44564 deadline: 1681606638008, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:18,012 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:18,012 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44548 deadline: 1681606638008, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:18,014 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:18,015 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44562 deadline: 1681606638012, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:18,184 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=185 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp/B/2ddc315b0b184c4fb52a9135c43dc968 2023-04-16 00:56:18,199 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=185 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp/C/432c54ce7abc4f13b6ca52483ff9189e 2023-04-16 00:56:18,203 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp/A/21025b18a78649a88ca998e1d8ebae26 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A/21025b18a78649a88ca998e1d8ebae26 2023-04-16 00:56:18,206 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A/21025b18a78649a88ca998e1d8ebae26, entries=150, sequenceid=185, filesize=25.3 K 2023-04-16 00:56:18,207 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp/B/2ddc315b0b184c4fb52a9135c43dc968 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/2ddc315b0b184c4fb52a9135c43dc968 2023-04-16 00:56:18,210 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/2ddc315b0b184c4fb52a9135c43dc968, entries=150, sequenceid=185, filesize=11.8 K 2023-04-16 00:56:18,211 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp/C/432c54ce7abc4f13b6ca52483ff9189e as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/432c54ce7abc4f13b6ca52483ff9189e 2023-04-16 00:56:18,214 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/432c54ce7abc4f13b6ca52483ff9189e, entries=150, sequenceid=185, filesize=11.8 K 2023-04-16 00:56:18,215 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~360.33 KB/368976, currentSize=67.09 KB/68700 for d538367277459b70f4a06235dfaacfec in 887ms, sequenceid=185, compaction requested=true 2023-04-16 00:56:18,215 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for d538367277459b70f4a06235dfaacfec: 2023-04-16 00:56:18,215 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. 2023-04-16 00:56:18,215 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:56:18,215 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:56:18,215 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:18,215 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:56:18,215 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:56:18,216 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:18,216 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:18,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:18,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:18,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:18,217 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:18,217 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:56:18,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:18,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:18,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:18,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:18,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:18,219 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:56:18,219 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3ff3a038[Count = 0] remaining members to acquire global barrier 2023-04-16 00:56:18,219 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:56:18,219 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:18,222 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:18,222 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:18,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:18,222 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:56:18,222 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:56:18,222 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:56:18,223 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:18,223 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:56:18,225 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:56:18,225 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:56:18,225 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:18,225 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:56:18,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:18,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:18,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:18,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:18,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:18,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:18,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:18,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:18,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:18,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:18,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:56:18,228 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:56:18,228 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:56:18,228 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:56:18,228 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:56:18,228 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:56:18,228 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:56:18,233 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:18,233 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:18,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:18,233 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:18,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:18,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:18,233 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:56:18,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:18,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:18,233 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:56:18,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:56:18,234 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:18,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:18,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:18,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:18,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:18,235 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:18,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:18,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:18,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:18,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:18,237 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:56:18,237 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:56:18,237 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:18,237 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:18,237 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:56:18,237 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:56:18,237 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:56:18,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:56:18,237 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:56:18,237 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:56:18,237 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:18,237 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:18,237 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:18,237 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:18,237 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:18,238 DEBUG [Thread-4138] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:56:18,238 DEBUG [Thread-4138] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:56:18,238 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:56:18,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:56:18,338 DEBUG [Thread-4138] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:56:18,338 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:56:18,341 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:56:18,342 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:56:18,342 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:56:18,342 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:56:18,342 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:56:18,342 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:56:18,342 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:56:18,343 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:18,343 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:18,344 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:56:18,344 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:18,344 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:18,344 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:56:18,344 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:56:18,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:56:18,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:18,345 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:18,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:56:18,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:18,345 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:56:18,346 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:56:18,346 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:56:18,346 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:56:18,346 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:56:18,346 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:56:18,346 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:56:18,346 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:56:18,346 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. 2023-04-16 00:56:18,346 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. started... 2023-04-16 00:56:18,346 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing d538367277459b70f4a06235dfaacfec 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-16 00:56:18,347 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d538367277459b70f4a06235dfaacfec, store=A 2023-04-16 00:56:18,347 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:56:18,347 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d538367277459b70f4a06235dfaacfec, store=B 2023-04-16 00:56:18,347 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:56:18,347 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d538367277459b70f4a06235dfaacfec, store=C 2023-04-16 00:56:18,347 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:56:18,360 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230416ba2c64382fec47b9a178d521ba6617fe to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230416ba2c64382fec47b9a178d521ba6617fe 2023-04-16 00:56:18,366 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=201, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp/A/3947ddc746d24d7e80c706bc115db731 2023-04-16 00:56:18,376 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=201 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp/B/eee4a4e7ffc1455fb34c55e7796870bb 2023-04-16 00:56:18,385 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=201 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp/C/0b150b353ab549cd98002fc17a4fb298 2023-04-16 00:56:18,389 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp/A/3947ddc746d24d7e80c706bc115db731 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A/3947ddc746d24d7e80c706bc115db731 2023-04-16 00:56:18,393 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A/3947ddc746d24d7e80c706bc115db731, entries=150, sequenceid=201, filesize=25.3 K 2023-04-16 00:56:18,394 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp/B/eee4a4e7ffc1455fb34c55e7796870bb as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/eee4a4e7ffc1455fb34c55e7796870bb 2023-04-16 00:56:18,397 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/eee4a4e7ffc1455fb34c55e7796870bb, entries=150, sequenceid=201, filesize=11.8 K 2023-04-16 00:56:18,397 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp/C/0b150b353ab549cd98002fc17a4fb298 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/0b150b353ab549cd98002fc17a4fb298 2023-04-16 00:56:18,401 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/0b150b353ab549cd98002fc17a4fb298, entries=150, sequenceid=201, filesize=11.8 K 2023-04-16 00:56:18,402 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=0 B/0 for d538367277459b70f4a06235dfaacfec in 56ms, sequenceid=201, compaction requested=true 2023-04-16 00:56:18,402 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for d538367277459b70f4a06235dfaacfec: 2023-04-16 00:56:18,402 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. 2023-04-16 00:56:18,402 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:56:18,402 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:56:18,402 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:18,402 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:56:18,402 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:56:18,404 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:18,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:18,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:18,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:18,404 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:18,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:18,405 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:18,405 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:56:18,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:18,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:18,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:18,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:18,406 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:56:18,406 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2462b6d0[Count = 0] remaining members to acquire global barrier 2023-04-16 00:56:18,406 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:56:18,407 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:18,407 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:18,408 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:18,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:18,408 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:56:18,408 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:56:18,408 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:56:18,408 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:18,408 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:56:18,409 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:18,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:18,409 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:56:18,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:18,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:18,409 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:56:18,409 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:56:18,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:18,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:18,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:18,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:18,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:18,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:18,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:18,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:56:18,412 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:56:18,412 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:56:18,412 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:56:18,412 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:56:18,412 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:56:18,412 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:56:18,414 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:18,414 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:56:18,414 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:18,414 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:18,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:18,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:18,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:18,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:18,415 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:56:18,415 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:18,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:18,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:56:18,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:18,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:18,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:18,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:18,416 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:18,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:18,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:18,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:18,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:18,422 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:18,422 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:56:18,422 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:18,422 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:56:18,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:56:18,422 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:56:18,422 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:56:18,422 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:56:18,422 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:56:18,422 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:56:18,422 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:18,423 DEBUG [Thread-4138] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:56:18,423 DEBUG [Thread-4138] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:56:18,423 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:18,423 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:18,423 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:18,423 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:18,423 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:56:18,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:56:18,446 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(9158): Flush requested on d538367277459b70f4a06235dfaacfec 2023-04-16 00:56:18,446 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing d538367277459b70f4a06235dfaacfec 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-16 00:56:18,447 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d538367277459b70f4a06235dfaacfec, store=A 2023-04-16 00:56:18,447 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:56:18,448 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d538367277459b70f4a06235dfaacfec, store=B 2023-04-16 00:56:18,448 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:56:18,448 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d538367277459b70f4a06235dfaacfec, store=C 2023-04-16 00:56:18,448 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:56:18,462 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023041686dd15117d8d4fb78a89136c80a1d6a3 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041686dd15117d8d4fb78a89136c80a1d6a3 2023-04-16 00:56:18,466 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=212, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp/A/d6467e9fff6b4b00b6f7f74a31c43aee 2023-04-16 00:56:18,486 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1681606578475, max timestamp=1681606578484Num uniques 859; 2023-04-16 00:56:18,486 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-16 00:56:18,486 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1681606578475, max timestamp=1681606578484Num uniques 859; 2023-04-16 00:56:18,486 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-16 00:56:18,486 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1681606578475, max timestamp=1681606578484Num uniques 859; 2023-04-16 00:56:18,486 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-16 00:56:18,518 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:18,519 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44548 deadline: 1681606638516, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:18,519 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:18,519 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44556 deadline: 1681606638517, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:18,522 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:18,522 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44564 deadline: 1681606638518, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:18,522 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:18,522 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44554 deadline: 1681606638519, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:18,522 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:18,522 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44562 deadline: 1681606638519, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:18,523 DEBUG [Thread-4138] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:56:18,523 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:56:18,525 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:56:18,528 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:56:18,528 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:56:18,528 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:56:18,528 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:56:18,528 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:56:18,528 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:56:18,528 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:18,528 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:18,535 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:56:18,535 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:18,535 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:56:18,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:56:18,535 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:18,535 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:56:18,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:18,535 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:18,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:56:18,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:18,536 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:56:18,536 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:56:18,536 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:56:18,536 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:56:18,537 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:56:18,537 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:56:18,540 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:56:18,540 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:56:18,540 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. 2023-04-16 00:56:18,540 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. started... 2023-04-16 00:56:18,540 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. as already flushing 2023-04-16 00:56:18,540 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. 2023-04-16 00:56:18,540 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:56:18,540 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:56:18,540 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:18,540 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:56:18,540 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:56:18,542 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:18,542 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:18,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:18,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:18,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:18,543 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:18,543 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:56:18,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:18,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:18,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:18,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:18,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:18,544 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:56:18,544 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6c8c3c60[Count = 0] remaining members to acquire global barrier 2023-04-16 00:56:18,544 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:56:18,544 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:18,545 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:18,545 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:18,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:18,546 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:56:18,546 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:56:18,546 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:56:18,546 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:18,546 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:56:18,548 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:18,548 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:56:18,548 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:56:18,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:18,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:18,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:18,548 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:56:18,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:18,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:18,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:18,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:18,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:18,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:18,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:18,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:56:18,550 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:56:18,550 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:56:18,550 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:56:18,550 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:56:18,550 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:56:18,550 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:56:18,552 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:18,552 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:56:18,552 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:18,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:18,553 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:56:18,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:56:18,553 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:18,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:18,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:18,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:18,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:18,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:18,553 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:18,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:18,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:18,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:18,555 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:18,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:18,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:18,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:18,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:18,557 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:56:18,557 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:56:18,557 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:56:18,557 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:56:18,557 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:56:18,557 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:18,558 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:18,558 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:18,558 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:18,558 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:56:18,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:56:18,557 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:56:18,558 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:18,558 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:18,558 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:18,558 DEBUG [Thread-4138] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:56:18,558 DEBUG [Thread-4138] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:56:18,558 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:56:18,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:56:18,624 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:18,624 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44556 deadline: 1681606638621, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:18,626 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:18,626 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44564 deadline: 1681606638623, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:18,626 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:18,626 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44554 deadline: 1681606638623, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:18,658 DEBUG [Thread-4138] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:56:18,659 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:56:18,660 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:56:18,661 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:56:18,662 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:56:18,662 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:56:18,662 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:56:18,662 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:56:18,662 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:56:18,662 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:18,663 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:18,664 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:56:18,664 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:18,664 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:56:18,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:56:18,664 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:18,664 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:56:18,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:18,665 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:18,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:56:18,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:18,665 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:56:18,665 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:56:18,665 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:56:18,665 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:56:18,665 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:56:18,665 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:56:18,666 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:56:18,666 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:56:18,666 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. 2023-04-16 00:56:18,666 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. started... 2023-04-16 00:56:18,666 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. as already flushing 2023-04-16 00:56:18,666 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. 2023-04-16 00:56:18,666 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:56:18,666 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:56:18,666 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:18,666 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:56:18,666 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:56:18,668 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:18,668 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:18,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:18,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:18,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:18,669 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:18,669 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:56:18,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:18,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:18,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:18,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:18,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:18,670 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:56:18,670 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@e8e684f[Count = 0] remaining members to acquire global barrier 2023-04-16 00:56:18,670 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:56:18,670 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:18,671 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:18,671 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:18,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:18,671 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:56:18,671 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:56:18,671 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:56:18,671 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:18,671 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:56:18,672 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:18,672 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:56:18,673 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:56:18,673 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:56:18,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:18,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:18,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:18,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:18,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:18,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:18,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:18,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:18,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:18,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:18,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:56:18,675 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:56:18,675 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:56:18,675 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:56:18,675 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:56:18,675 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:56:18,675 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:56:18,677 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:18,677 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:18,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:18,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:18,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:18,677 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:18,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:18,677 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:56:18,678 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:56:18,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:18,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:56:18,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:18,678 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:18,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:18,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:18,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:18,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:18,679 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:18,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:18,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:18,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:18,682 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:18,682 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:18,682 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:18,682 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:18,682 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:18,682 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:18,682 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:56:18,682 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:56:18,682 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:56:18,683 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:56:18,682 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:18,682 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:56:18,682 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:56:18,683 DEBUG [Thread-4138] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:56:18,682 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:56:18,683 DEBUG [Thread-4138] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:56:18,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:56:18,683 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:56:18,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:56:18,783 DEBUG [Thread-4138] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:56:18,784 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:56:18,785 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:56:18,786 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:56:18,786 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:56:18,786 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:56:18,786 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:56:18,786 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:56:18,786 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:56:18,787 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:18,787 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:18,788 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:56:18,788 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:18,788 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:56:18,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:56:18,788 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:18,788 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:56:18,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:18,789 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:18,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:56:18,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:18,789 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:56:18,789 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:56:18,789 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:56:18,789 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:56:18,790 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:56:18,790 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:56:18,790 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:56:18,790 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. 2023-04-16 00:56:18,790 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. started... 2023-04-16 00:56:18,790 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:56:18,790 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. as already flushing 2023-04-16 00:56:18,790 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. 2023-04-16 00:56:18,790 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:56:18,790 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:56:18,790 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:18,790 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:56:18,790 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:56:18,791 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:18,791 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:18,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:18,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:18,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:18,792 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:18,792 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:56:18,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:18,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:18,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:18,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:18,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:18,793 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:56:18,793 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2c5eebc6[Count = 0] remaining members to acquire global barrier 2023-04-16 00:56:18,793 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:56:18,793 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:18,794 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:18,794 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:18,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:18,794 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:56:18,795 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:56:18,794 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:18,795 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:56:18,795 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:56:18,797 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:18,797 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:56:18,797 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:56:18,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:18,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:18,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:18,797 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:56:18,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:18,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:18,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:18,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:18,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:18,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:18,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:18,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:56:18,799 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:56:18,799 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:56:18,800 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:56:18,800 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:56:18,800 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:56:18,800 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:56:18,803 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:18,803 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:18,803 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:18,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:18,803 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:56:18,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:18,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:18,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:18,803 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:56:18,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:56:18,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:18,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:18,804 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:18,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:18,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:18,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:18,805 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:18,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:18,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:18,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:18,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:18,807 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:56:18,807 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:56:18,807 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:56:18,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:56:18,807 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:56:18,807 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:18,807 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:18,807 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:18,807 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:18,807 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:56:18,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:56:18,807 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:18,807 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:18,807 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:18,807 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:56:18,807 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:56:18,808 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:56:18,808 DEBUG [Thread-4138] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:56:18,808 DEBUG [Thread-4138] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:56:18,828 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:18,829 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44556 deadline: 1681606638825, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:18,829 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:18,829 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44564 deadline: 1681606638827, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:18,830 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:18,830 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44554 deadline: 1681606638828, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:18,881 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=212 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp/B/f242a0f93a004e8e87a5b468b65420cd 2023-04-16 00:56:18,908 DEBUG [Thread-4138] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:56:18,909 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:56:18,910 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:56:18,911 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:56:18,911 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:56:18,911 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:56:18,911 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:56:18,911 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:56:18,911 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:56:18,912 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:18,912 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:18,914 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:18,914 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:56:18,914 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:56:18,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:56:18,914 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:18,914 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:56:18,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:18,914 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:18,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:56:18,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:18,915 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:56:18,915 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:56:18,915 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:56:18,915 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:56:18,915 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:56:18,915 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:56:18,915 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:56:18,915 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. 2023-04-16 00:56:18,915 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. started... 2023-04-16 00:56:18,915 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:56:18,916 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. as already flushing 2023-04-16 00:56:18,916 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. 2023-04-16 00:56:18,916 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:56:18,916 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:56:18,916 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:18,916 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:56:18,916 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:56:18,917 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:18,917 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:18,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:18,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:18,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:18,918 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:18,918 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:56:18,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:18,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:18,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:18,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:18,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:18,919 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:56:18,919 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1d3b1cab[Count = 0] remaining members to acquire global barrier 2023-04-16 00:56:18,919 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:56:18,919 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:18,920 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:18,920 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:18,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:18,920 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:18,920 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:56:18,920 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:56:18,920 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:56:18,921 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:56:18,922 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:18,922 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:56:18,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:18,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:18,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:18,922 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:56:18,922 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:56:18,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:18,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:18,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:18,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:18,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:18,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:18,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:18,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:56:18,924 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:56:18,924 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:56:18,925 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:56:18,925 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:56:18,925 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:56:18,925 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:56:18,927 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:18,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:18,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:18,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:18,927 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:18,927 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:56:18,927 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:18,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:18,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:18,928 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:56:18,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:56:18,928 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:18,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:18,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:18,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:18,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:18,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:18,929 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:18,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:18,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:18,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:18,932 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:56:18,932 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:56:18,932 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:18,932 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:18,932 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:18,932 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:18,932 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:18,933 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:18,933 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:18,933 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:56:18,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:56:18,933 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:56:18,933 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:56:18,933 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:56:18,933 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:56:18,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:56:18,933 DEBUG [Thread-4138] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:56:18,933 DEBUG [Thread-4138] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:56:19,033 DEBUG [Thread-4138] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:56:19,034 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:56:19,035 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:56:19,036 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:56:19,037 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:56:19,037 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:56:19,037 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:56:19,037 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:56:19,037 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:56:19,037 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:19,037 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:19,038 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:56:19,038 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:19,038 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:56:19,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:56:19,039 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:19,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:19,039 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:56:19,039 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:19,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:56:19,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:19,039 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:56:19,039 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:56:19,040 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:56:19,040 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:56:19,040 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:56:19,040 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:56:19,040 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:56:19,040 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. 2023-04-16 00:56:19,040 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. started... 2023-04-16 00:56:19,040 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:56:19,040 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. as already flushing 2023-04-16 00:56:19,040 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. 2023-04-16 00:56:19,040 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:56:19,040 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:56:19,040 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:19,040 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:56:19,040 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:56:19,041 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:19,041 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:19,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:19,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:19,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:19,042 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:19,042 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:56:19,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:19,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:19,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:19,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:19,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:19,043 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:56:19,043 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@10d94c50[Count = 0] remaining members to acquire global barrier 2023-04-16 00:56:19,043 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:56:19,043 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:19,049 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:19,049 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:19,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:19,049 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:56:19,049 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:56:19,049 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:19,049 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:56:19,049 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:56:19,050 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:56:19,050 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:19,050 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:56:19,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:19,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:19,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:19,051 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:56:19,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:19,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:19,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:19,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:19,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:19,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:19,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:19,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:56:19,053 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:56:19,053 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:56:19,053 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:56:19,053 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:56:19,053 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:56:19,053 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:56:19,054 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:19,054 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:19,054 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:19,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:19,054 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:56:19,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:19,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:19,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:19,055 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:56:19,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:56:19,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:19,055 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:19,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:19,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:19,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:19,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:19,056 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:19,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:19,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:19,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:19,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:19,058 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:19,058 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:19,058 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:19,058 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:56:19,058 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:56:19,058 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:56:19,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:56:19,058 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:56:19,058 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:56:19,058 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:56:19,058 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:56:19,058 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:19,058 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:56:19,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:56:19,059 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:19,059 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:19,059 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:19,059 DEBUG [Thread-4138] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:56:19,059 DEBUG [Thread-4138] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:56:19,133 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:19,134 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44564 deadline: 1681606639131, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:19,134 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:19,134 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44556 deadline: 1681606639131, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:19,135 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:19,135 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44554 deadline: 1681606639133, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:19,159 DEBUG [Thread-4138] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:56:19,159 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:56:19,161 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:56:19,162 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:56:19,162 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:56:19,162 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:56:19,162 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:56:19,162 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:56:19,162 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:56:19,163 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:19,163 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:19,165 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:19,165 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:56:19,165 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:56:19,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:56:19,165 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:19,165 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:56:19,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:19,166 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:19,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:56:19,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:19,166 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:56:19,166 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:56:19,166 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:56:19,166 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:56:19,166 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:56:19,166 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:56:19,167 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:56:19,167 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:56:19,167 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. 2023-04-16 00:56:19,167 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. started... 2023-04-16 00:56:19,167 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. as already flushing 2023-04-16 00:56:19,167 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. 2023-04-16 00:56:19,167 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:56:19,167 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:56:19,167 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:19,167 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:56:19,167 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:56:19,169 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:19,169 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:19,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:19,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:19,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:19,169 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:19,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:19,169 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:56:19,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:19,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:19,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:19,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:19,170 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:56:19,170 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6de88214[Count = 0] remaining members to acquire global barrier 2023-04-16 00:56:19,170 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:56:19,170 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:19,171 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:19,171 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:19,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:19,171 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:56:19,172 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:56:19,172 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:56:19,172 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:19,172 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:56:19,173 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:56:19,173 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:56:19,173 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:19,173 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:56:19,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:19,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:19,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:19,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:19,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:19,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:19,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:19,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:19,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:19,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:19,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:56:19,176 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:56:19,176 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:56:19,176 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:56:19,176 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:56:19,176 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:56:19,176 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:56:19,179 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:19,179 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:19,179 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:56:19,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:19,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:19,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:19,179 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:19,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:19,179 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:19,179 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:56:19,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:56:19,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:19,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:19,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:19,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:19,181 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:19,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:19,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:19,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:19,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:19,183 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:19,183 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:19,183 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:19,183 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:19,183 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:19,183 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:56:19,183 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:56:19,183 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:56:19,183 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:56:19,183 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:19,184 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:19,183 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:56:19,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:56:19,183 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:56:19,184 DEBUG [Thread-4138] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:56:19,184 DEBUG [Thread-4138] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:56:19,184 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:56:19,184 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:56:19,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:56:19,284 DEBUG [Thread-4138] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:56:19,285 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:56:19,286 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:56:19,290 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:56:19,290 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:56:19,290 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:56:19,290 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:56:19,290 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:56:19,290 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:56:19,290 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=212 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp/C/b3f9b6a3d4514fac89ae7134e1859c8a 2023-04-16 00:56:19,291 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:19,291 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:19,294 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp/A/d6467e9fff6b4b00b6f7f74a31c43aee as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A/d6467e9fff6b4b00b6f7f74a31c43aee 2023-04-16 00:56:19,295 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:56:19,295 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:19,295 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:56:19,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:56:19,295 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:19,295 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:56:19,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:19,296 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:19,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:56:19,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:19,296 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:56:19,296 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:56:19,296 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:56:19,297 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:56:19,297 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:56:19,297 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:56:19,297 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:56:19,297 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:56:19,297 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. 2023-04-16 00:56:19,297 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. started... 2023-04-16 00:56:19,297 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. as already flushing 2023-04-16 00:56:19,297 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. 2023-04-16 00:56:19,297 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:56:19,297 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:56:19,297 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:19,297 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:56:19,297 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:56:19,297 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A/d6467e9fff6b4b00b6f7f74a31c43aee, entries=200, sequenceid=212, filesize=32.1 K 2023-04-16 00:56:19,298 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp/B/f242a0f93a004e8e87a5b468b65420cd as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/f242a0f93a004e8e87a5b468b65420cd 2023-04-16 00:56:19,300 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:19,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:19,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:19,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:19,300 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:19,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:19,300 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:19,300 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:56:19,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:19,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:19,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:19,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:19,302 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:56:19,302 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@36b354b4[Count = 0] remaining members to acquire global barrier 2023-04-16 00:56:19,302 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:56:19,302 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:19,302 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/f242a0f93a004e8e87a5b468b65420cd, entries=150, sequenceid=212, filesize=11.8 K 2023-04-16 00:56:19,302 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp/C/b3f9b6a3d4514fac89ae7134e1859c8a as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/b3f9b6a3d4514fac89ae7134e1859c8a 2023-04-16 00:56:19,303 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:19,303 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:19,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:19,303 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:56:19,303 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:56:19,303 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:56:19,303 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:19,303 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:56:19,305 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:19,305 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:56:19,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:19,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:19,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:19,305 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:56:19,305 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:56:19,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:19,306 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/b3f9b6a3d4514fac89ae7134e1859c8a, entries=150, sequenceid=212, filesize=11.8 K 2023-04-16 00:56:19,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:19,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:19,306 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=134.18 KB/137400 for d538367277459b70f4a06235dfaacfec in 860ms, sequenceid=212, compaction requested=true 2023-04-16 00:56:19,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:19,306 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for d538367277459b70f4a06235dfaacfec: 2023-04-16 00:56:19,306 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 00:56:19,307 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-16 00:56:19,307 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-16 00:56:19,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:19,307 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 00:56:19,307 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 00:56:19,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:19,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:19,309 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48486 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-16 00:56:19,309 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HStore(1912): d538367277459b70f4a06235dfaacfec/B is initiating minor compaction (all files) 2023-04-16 00:56:19,309 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HRegion(2259): Starting compaction of d538367277459b70f4a06235dfaacfec/B in TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. 2023-04-16 00:56:19,309 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/ee3926f1cb034130a58d1315408ab2f1, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/2ddc315b0b184c4fb52a9135c43dc968, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/eee4a4e7ffc1455fb34c55e7796870bb, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/f242a0f93a004e8e87a5b468b65420cd] into tmpdir=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp, totalSize=47.3 K 2023-04-16 00:56:19,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:56:19,309 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 197464 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-16 00:56:19,309 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:56:19,310 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:56:19,310 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:56:19,310 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:56:19,310 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:56:19,309 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:56:19,310 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HStore(1912): d538367277459b70f4a06235dfaacfec/A is initiating minor compaction (all files) 2023-04-16 00:56:19,310 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of d538367277459b70f4a06235dfaacfec/A in TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. 2023-04-16 00:56:19,310 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A/a43dd4d44481416c85d473f9b3e6639f, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A/25122e76a3ac47b8accde73fc4baa0c1, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A/1d92c3f646be4e918c53aa9745ef7bb5, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A/21025b18a78649a88ca998e1d8ebae26, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A/3947ddc746d24d7e80c706bc115db731, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A/d6467e9fff6b4b00b6f7f74a31c43aee] into tmpdir=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp, totalSize=192.8 K 2023-04-16 00:56:19,310 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.Compactor(207): Compacting a43dd4d44481416c85d473f9b3e6639f, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=129, earliestPutTs=1681606576433 2023-04-16 00:56:19,311 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting ee3926f1cb034130a58d1315408ab2f1, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=150, earliestPutTs=1681606577221 2023-04-16 00:56:19,311 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting 2ddc315b0b184c4fb52a9135c43dc968, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=185, earliestPutTs=1681606577264 2023-04-16 00:56:19,311 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.Compactor(207): Compacting 25122e76a3ac47b8accde73fc4baa0c1, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=139, earliestPutTs=1681606576552 2023-04-16 00:56:19,312 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting eee4a4e7ffc1455fb34c55e7796870bb, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=201, earliestPutTs=1681606577360 2023-04-16 00:56:19,312 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.Compactor(207): Compacting 1d92c3f646be4e918c53aa9745ef7bb5, keycount=400, bloomtype=ROW, size=59.4 K, encoding=NONE, compression=NONE, seqNum=150, earliestPutTs=1681606577212 2023-04-16 00:56:19,312 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:19,312 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:56:19,312 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:19,312 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:19,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:19,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:19,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:19,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:19,312 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.Compactor(207): Compacting 21025b18a78649a88ca998e1d8ebae26, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=185, earliestPutTs=1681606577264 2023-04-16 00:56:19,313 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:56:19,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:56:19,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:19,313 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:19,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:19,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:19,313 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting f242a0f93a004e8e87a5b468b65420cd, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=212, earliestPutTs=1681606578421 2023-04-16 00:56:19,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:19,314 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.Compactor(207): Compacting 3947ddc746d24d7e80c706bc115db731, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=201, earliestPutTs=1681606577360 2023-04-16 00:56:19,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:19,314 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.Compactor(207): Compacting d6467e9fff6b4b00b6f7f74a31c43aee, keycount=200, bloomtype=ROW, size=32.1 K, encoding=NONE, compression=NONE, seqNum=212, earliestPutTs=1681606578421 2023-04-16 00:56:19,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:19,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:19,316 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:19,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:19,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:19,319 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:19,320 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:19,320 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:19,320 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:19,320 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:19,320 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:19,320 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:19,320 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:56:19,320 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:56:19,320 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:56:19,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:56:19,320 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:56:19,320 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:56:19,320 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:56:19,320 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:56:19,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:56:19,321 DEBUG [Thread-4138] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:56:19,321 DEBUG [Thread-4138] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:56:19,324 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] throttle.PressureAwareThroughputController(145): d538367277459b70f4a06235dfaacfec#B#compaction#596 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-16 00:56:19,337 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] throttle.PressureAwareThroughputController(145): d538367277459b70f4a06235dfaacfec#A#compaction#597 average throughput is 2.19 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-16 00:56:19,337 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp/B/2c25715a76ec4aa9bbe7cdaa21f15cf5 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/2c25715a76ec4aa9bbe7cdaa21f15cf5 2023-04-16 00:56:19,343 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in d538367277459b70f4a06235dfaacfec/B of d538367277459b70f4a06235dfaacfec into 2c25715a76ec4aa9bbe7cdaa21f15cf5(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 00:56:19,343 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for d538367277459b70f4a06235dfaacfec: 2023-04-16 00:56:19,343 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec., storeName=d538367277459b70f4a06235dfaacfec/B, priority=12, startTime=1681606579307; duration=0sec 2023-04-16 00:56:19,343 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 00:56:19,343 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-16 00:56:19,344 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48486 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-16 00:56:19,345 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HStore(1912): d538367277459b70f4a06235dfaacfec/C is initiating minor compaction (all files) 2023-04-16 00:56:19,345 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HRegion(2259): Starting compaction of d538367277459b70f4a06235dfaacfec/C in TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. 2023-04-16 00:56:19,345 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/259d737a359841719c910c2f0e8fc933, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/432c54ce7abc4f13b6ca52483ff9189e, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/0b150b353ab549cd98002fc17a4fb298, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/b3f9b6a3d4514fac89ae7134e1859c8a] into tmpdir=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp, totalSize=47.3 K 2023-04-16 00:56:19,345 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting 259d737a359841719c910c2f0e8fc933, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=150, earliestPutTs=1681606577221 2023-04-16 00:56:19,346 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting 432c54ce7abc4f13b6ca52483ff9189e, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=185, earliestPutTs=1681606577264 2023-04-16 00:56:19,346 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting 0b150b353ab549cd98002fc17a4fb298, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=201, earliestPutTs=1681606577360 2023-04-16 00:56:19,347 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting b3f9b6a3d4514fac89ae7134e1859c8a, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=212, earliestPutTs=1681606578421 2023-04-16 00:56:19,353 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] throttle.PressureAwareThroughputController(145): d538367277459b70f4a06235dfaacfec#C#compaction#598 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-16 00:56:19,360 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp/C/0ab1f2532b1c4d8bbab20bddceec9d47 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/0ab1f2532b1c4d8bbab20bddceec9d47 2023-04-16 00:56:19,363 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in d538367277459b70f4a06235dfaacfec/C of d538367277459b70f4a06235dfaacfec into 0ab1f2532b1c4d8bbab20bddceec9d47(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 00:56:19,363 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for d538367277459b70f4a06235dfaacfec: 2023-04-16 00:56:19,363 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec., storeName=d538367277459b70f4a06235dfaacfec/C, priority=12, startTime=1681606579307; duration=0sec 2023-04-16 00:56:19,364 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 00:56:19,421 DEBUG [Thread-4138] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:56:19,421 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:56:19,423 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:56:19,424 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:56:19,424 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:56:19,424 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:56:19,424 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:56:19,424 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:56:19,424 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:56:19,425 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:19,425 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:19,426 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:56:19,426 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:19,426 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:56:19,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:56:19,426 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:19,426 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:56:19,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:19,426 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:19,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:56:19,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:19,427 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:56:19,427 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:56:19,427 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:56:19,427 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:56:19,427 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:56:19,427 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:56:19,427 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:56:19,427 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:56:19,427 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. 2023-04-16 00:56:19,427 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. started... 2023-04-16 00:56:19,428 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing d538367277459b70f4a06235dfaacfec 3/3 column families, dataSize=134.18 KB heapSize=380.30 KB 2023-04-16 00:56:19,428 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d538367277459b70f4a06235dfaacfec, store=A 2023-04-16 00:56:19,428 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 00:56:19,428 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d538367277459b70f4a06235dfaacfec, store=B 2023-04-16 00:56:19,428 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 00:56:19,428 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d538367277459b70f4a06235dfaacfec, store=C 2023-04-16 00:56:19,428 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 00:56:19,444 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230416e1f6be038a26467e9e57a9ca8c1e4652 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230416e1f6be038a26467e9e57a9ca8c1e4652 2023-04-16 00:56:19,449 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=250, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp/A/e5ac83fbb7684a0ca28b839dfff5231e 2023-04-16 00:56:19,477 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=250 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp/B/98d55edf116f4da28ce9d4768d606e00 2023-04-16 00:56:19,530 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(9158): Flush requested on d538367277459b70f4a06235dfaacfec 2023-04-16 00:56:19,530 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. as already flushing 2023-04-16 00:56:19,547 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:19,547 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44548 deadline: 1681606639544, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:19,547 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:19,548 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44562 deadline: 1681606639544, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:19,637 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:19,637 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44556 deadline: 1681606639636, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:19,640 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:19,640 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44564 deadline: 1681606639638, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:19,640 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:19,640 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44554 deadline: 1681606639638, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:19,650 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:19,650 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44548 deadline: 1681606639648, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:19,650 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:19,650 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44562 deadline: 1681606639648, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:19,763 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp/A/cd94a09e5f6e4122ac967486e097084c as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A/cd94a09e5f6e4122ac967486e097084c 2023-04-16 00:56:19,767 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in d538367277459b70f4a06235dfaacfec/A of d538367277459b70f4a06235dfaacfec into cd94a09e5f6e4122ac967486e097084c(size=25.7 K), total size for store is 25.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 00:56:19,767 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for d538367277459b70f4a06235dfaacfec: 2023-04-16 00:56:19,767 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec., storeName=d538367277459b70f4a06235dfaacfec/A, priority=10, startTime=1681606579306; duration=0sec 2023-04-16 00:56:19,767 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 00:56:19,854 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:19,854 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44548 deadline: 1681606639851, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:19,854 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:19,854 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44562 deadline: 1681606639852, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:19,887 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=250 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp/C/8546acaf4e1e47f0ba2fb98ced02e994 2023-04-16 00:56:19,891 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp/A/e5ac83fbb7684a0ca28b839dfff5231e as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A/e5ac83fbb7684a0ca28b839dfff5231e 2023-04-16 00:56:19,894 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A/e5ac83fbb7684a0ca28b839dfff5231e, entries=150, sequenceid=250, filesize=25.3 K 2023-04-16 00:56:19,895 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp/B/98d55edf116f4da28ce9d4768d606e00 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/98d55edf116f4da28ce9d4768d606e00 2023-04-16 00:56:19,898 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/98d55edf116f4da28ce9d4768d606e00, entries=150, sequenceid=250, filesize=11.8 K 2023-04-16 00:56:19,898 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp/C/8546acaf4e1e47f0ba2fb98ced02e994 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/8546acaf4e1e47f0ba2fb98ced02e994 2023-04-16 00:56:19,901 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/8546acaf4e1e47f0ba2fb98ced02e994, entries=150, sequenceid=250, filesize=11.8 K 2023-04-16 00:56:19,902 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~380.25 KB/389376, currentSize=46.96 KB/48090 for d538367277459b70f4a06235dfaacfec in 474ms, sequenceid=250, compaction requested=false 2023-04-16 00:56:19,902 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for d538367277459b70f4a06235dfaacfec: 2023-04-16 00:56:19,902 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. 2023-04-16 00:56:19,902 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:56:19,902 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:56:19,902 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:19,902 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:56:19,902 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:56:19,903 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:19,903 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:19,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:19,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:19,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:19,904 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:19,904 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:56:19,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:19,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:19,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:19,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:19,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:19,905 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:56:19,905 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@73a8c88d[Count = 0] remaining members to acquire global barrier 2023-04-16 00:56:19,905 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:56:19,905 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:19,906 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:19,906 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:19,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:19,907 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:56:19,907 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:56:19,907 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:56:19,907 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:19,907 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:56:19,909 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:19,909 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:56:19,909 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:56:19,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:19,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:19,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:19,909 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:56:19,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:19,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:19,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:19,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:19,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:19,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:19,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:19,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:56:19,911 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:56:19,911 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:56:19,911 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:56:19,911 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:56:19,911 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:56:19,912 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:56:19,913 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:19,913 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:56:19,913 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:19,914 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:19,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:19,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:19,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:19,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:19,914 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:56:19,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:56:19,914 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:19,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:19,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:19,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:19,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:19,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:19,915 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:19,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:19,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:19,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:19,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:19,917 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:19,917 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:19,917 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:19,917 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:56:19,917 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:56:19,917 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:19,918 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:56:19,918 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:56:19,918 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:56:19,918 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:19,918 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:19,918 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:19,918 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:56:19,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:56:19,918 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:56:19,918 DEBUG [Thread-4138] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:56:19,918 DEBUG [Thread-4138] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:56:19,918 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:56:19,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:56:20,018 DEBUG [Thread-4138] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:56:20,019 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:56:20,020 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:56:20,021 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:56:20,021 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:56:20,021 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:56:20,021 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:56:20,021 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:56:20,021 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:56:20,022 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:20,022 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:20,023 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:56:20,023 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:20,023 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:56:20,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:56:20,023 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:20,023 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:56:20,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:20,024 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:20,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:56:20,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:20,024 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:56:20,024 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:56:20,024 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:56:20,024 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:56:20,024 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:56:20,024 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:56:20,025 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:56:20,025 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:56:20,025 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. 2023-04-16 00:56:20,025 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. started... 2023-04-16 00:56:20,025 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing d538367277459b70f4a06235dfaacfec 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-16 00:56:20,025 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d538367277459b70f4a06235dfaacfec, store=A 2023-04-16 00:56:20,025 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:56:20,025 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d538367277459b70f4a06235dfaacfec, store=B 2023-04-16 00:56:20,025 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:56:20,025 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d538367277459b70f4a06235dfaacfec, store=C 2023-04-16 00:56:20,025 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:56:20,158 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(9158): Flush requested on d538367277459b70f4a06235dfaacfec 2023-04-16 00:56:20,158 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. as already flushing 2023-04-16 00:56:20,216 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681606580202, max timestamp=1681606580214Num uniques 859; 2023-04-16 00:56:20,216 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-16 00:56:20,216 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681606580202, max timestamp=1681606580214Num uniques 859; 2023-04-16 00:56:20,216 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681606580202, max timestamp=1681606580214Num uniques 859; 2023-04-16 00:56:20,216 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-16 00:56:20,216 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-16 00:56:20,249 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:20,249 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44562 deadline: 1681606640245, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:20,251 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:20,252 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44548 deadline: 1681606640249, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:20,350 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:20,351 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44562 deadline: 1681606640350, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:20,354 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:20,355 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44548 deadline: 1681606640353, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:20,438 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304160afcc0ca4dc5474895e4312ebe5c67eb to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304160afcc0ca4dc5474895e4312ebe5c67eb 2023-04-16 00:56:20,443 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=261, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp/A/cdd41307649a455fbb20b175cacaec3c 2023-04-16 00:56:20,454 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=261 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp/B/08ad153cb7df419da541f7ba065dd7e3 2023-04-16 00:56:20,464 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=261 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp/C/e645f6b6bf834a8fab0c6977593479d9 2023-04-16 00:56:20,468 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp/A/cdd41307649a455fbb20b175cacaec3c as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A/cdd41307649a455fbb20b175cacaec3c 2023-04-16 00:56:20,471 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A/cdd41307649a455fbb20b175cacaec3c, entries=150, sequenceid=261, filesize=25.4 K 2023-04-16 00:56:20,472 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp/B/08ad153cb7df419da541f7ba065dd7e3 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/08ad153cb7df419da541f7ba065dd7e3 2023-04-16 00:56:20,476 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/08ad153cb7df419da541f7ba065dd7e3, entries=150, sequenceid=261, filesize=11.9 K 2023-04-16 00:56:20,477 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp/C/e645f6b6bf834a8fab0c6977593479d9 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/e645f6b6bf834a8fab0c6977593479d9 2023-04-16 00:56:20,482 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/e645f6b6bf834a8fab0c6977593479d9, entries=150, sequenceid=261, filesize=11.9 K 2023-04-16 00:56:20,482 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for d538367277459b70f4a06235dfaacfec in 457ms, sequenceid=261, compaction requested=true 2023-04-16 00:56:20,483 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for d538367277459b70f4a06235dfaacfec: 2023-04-16 00:56:20,483 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. 2023-04-16 00:56:20,483 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:56:20,483 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:56:20,483 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:20,483 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:56:20,483 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:56:20,486 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:20,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:20,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:20,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:20,486 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:20,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:20,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:20,486 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:20,487 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:56:20,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:20,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:20,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:20,488 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:56:20,488 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2949a164[Count = 0] remaining members to acquire global barrier 2023-04-16 00:56:20,488 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:56:20,488 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:20,489 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:20,489 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:20,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:20,490 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:56:20,490 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:56:20,490 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:56:20,491 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:20,491 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:56:20,492 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:56:20,492 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:56:20,492 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:20,492 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:56:20,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:20,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:20,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:20,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:20,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:20,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:20,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:20,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:20,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:20,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:20,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:56:20,496 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:56:20,496 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:56:20,496 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:56:20,496 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:56:20,496 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:56:20,496 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:56:20,498 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:20,498 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:56:20,498 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:20,498 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:20,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:20,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:20,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:20,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:20,498 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:56:20,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:56:20,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:20,499 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:20,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:20,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:20,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:20,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:20,500 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:20,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:20,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:20,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:20,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:20,503 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:20,503 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:56:20,504 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:20,504 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:20,504 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:20,504 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:56:20,504 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:56:20,504 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:56:20,504 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:56:20,504 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:56:20,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:56:20,504 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:56:20,504 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:20,504 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:20,504 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:20,504 DEBUG [Thread-4138] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:56:20,504 DEBUG [Thread-4138] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:56:20,505 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:56:20,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:56:20,554 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(9158): Flush requested on d538367277459b70f4a06235dfaacfec 2023-04-16 00:56:20,554 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing d538367277459b70f4a06235dfaacfec 3/3 column families, dataSize=140.89 KB heapSize=405.61 KB 2023-04-16 00:56:20,555 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d538367277459b70f4a06235dfaacfec, store=A 2023-04-16 00:56:20,555 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-16 00:56:20,555 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d538367277459b70f4a06235dfaacfec, store=B 2023-04-16 00:56:20,555 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-16 00:56:20,555 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d538367277459b70f4a06235dfaacfec, store=C 2023-04-16 00:56:20,555 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-16 00:56:20,567 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:20,567 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44548 deadline: 1681606640564, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:20,570 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:20,570 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44562 deadline: 1681606640567, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:20,605 DEBUG [Thread-4138] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:56:20,605 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:56:20,606 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:56:20,607 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:56:20,608 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:56:20,608 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:56:20,608 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:56:20,608 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:56:20,608 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:56:20,608 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:20,608 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:20,609 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:56:20,609 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:56:20,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:56:20,610 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:20,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:20,610 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:20,610 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:56:20,610 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:20,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:56:20,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:20,610 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:56:20,611 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:56:20,611 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:56:20,611 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:56:20,611 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:56:20,611 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:56:20,611 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:56:20,611 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:56:20,611 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. 2023-04-16 00:56:20,611 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. started... 2023-04-16 00:56:20,611 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. as already flushing 2023-04-16 00:56:20,611 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. 2023-04-16 00:56:20,611 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:56:20,611 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:56:20,611 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:20,612 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:56:20,612 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:56:20,613 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:20,613 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:20,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:20,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:20,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:20,613 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:20,613 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:56:20,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:20,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:20,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:20,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:20,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:20,615 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:56:20,615 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@986e7b[Count = 0] remaining members to acquire global barrier 2023-04-16 00:56:20,615 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:56:20,615 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:20,620 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:20,620 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:20,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:20,620 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:56:20,620 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:56:20,620 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:56:20,620 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:20,620 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:56:20,622 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:20,622 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:56:20,622 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:56:20,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:20,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:20,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:20,622 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:56:20,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:20,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:20,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:20,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:20,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:20,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:20,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:20,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:56:20,624 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:56:20,624 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:56:20,624 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:56:20,624 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:56:20,624 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:56:20,624 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:56:20,626 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:20,626 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:56:20,626 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:20,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:20,626 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:20,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:20,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:20,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:20,626 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:56:20,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:56:20,626 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:20,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:20,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:20,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:20,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:20,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:20,628 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:20,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:20,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:20,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:20,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:20,630 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:20,630 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:20,630 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:20,631 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:20,631 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:20,631 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:56:20,631 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:20,631 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:20,631 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:56:20,631 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:56:20,631 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:56:20,631 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:56:20,631 DEBUG [Thread-4138] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:56:20,631 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:56:20,631 DEBUG [Thread-4138] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:56:20,631 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:56:20,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:56:20,631 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:56:20,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:56:20,648 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:20,648 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44556 deadline: 1681606640645, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:20,650 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:20,650 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44554 deadline: 1681606640648, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:20,653 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:20,653 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44564 deadline: 1681606640650, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:20,671 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:20,672 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44548 deadline: 1681606640668, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:20,672 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:20,673 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44562 deadline: 1681606640671, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:20,731 DEBUG [Thread-4138] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:56:20,732 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:56:20,733 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:56:20,734 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:56:20,735 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:56:20,735 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:56:20,735 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:56:20,735 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:56:20,735 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:56:20,735 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:20,736 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:20,737 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:20,737 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:56:20,737 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:56:20,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:56:20,737 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:20,737 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:56:20,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:20,738 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:20,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:56:20,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:20,738 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:56:20,738 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:56:20,738 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:56:20,738 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:56:20,738 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:56:20,738 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:56:20,739 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:56:20,739 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:56:20,739 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. 2023-04-16 00:56:20,739 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. started... 2023-04-16 00:56:20,739 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. as already flushing 2023-04-16 00:56:20,739 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. 2023-04-16 00:56:20,739 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:56:20,739 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:56:20,739 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:20,739 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:56:20,739 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:56:20,740 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:20,740 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:20,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:20,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:20,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:20,741 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:20,741 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:56:20,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:20,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:20,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:20,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:20,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:20,742 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:56:20,742 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2c229c28[Count = 0] remaining members to acquire global barrier 2023-04-16 00:56:20,742 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:56:20,742 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:20,743 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:20,744 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:20,744 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:20,744 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:56:20,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:20,744 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:56:20,744 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:56:20,744 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:56:20,746 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:20,746 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:56:20,746 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:56:20,746 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:56:20,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:20,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:20,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:20,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:20,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:20,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:20,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:20,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:20,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:20,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:20,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:56:20,749 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:56:20,749 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:56:20,749 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:56:20,749 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:56:20,749 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:56:20,749 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:56:20,750 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:20,750 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:20,750 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:20,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:20,750 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:56:20,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:20,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:20,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:20,751 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:56:20,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:56:20,751 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:20,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:20,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:20,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:20,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:20,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:20,753 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:20,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:20,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:20,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:20,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:20,755 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:56:20,755 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:20,755 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:56:20,755 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:56:20,755 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:56:20,755 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:56:20,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:56:20,755 DEBUG [Thread-4138] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:56:20,755 DEBUG [Thread-4138] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:56:20,755 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:56:20,755 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:20,756 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:20,756 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:20,756 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:20,756 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:20,756 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:20,756 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:56:20,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:56:20,856 DEBUG [Thread-4138] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:56:20,856 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:56:20,858 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:56:20,859 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:56:20,859 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:56:20,859 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:56:20,859 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:56:20,859 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:56:20,859 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:56:20,859 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:20,859 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:20,861 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:56:20,861 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:20,861 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:56:20,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:56:20,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:20,861 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:20,862 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:56:20,862 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:20,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:56:20,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:20,862 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:56:20,862 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:56:20,862 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:56:20,862 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:56:20,862 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:56:20,863 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:56:20,863 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:56:20,863 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:56:20,863 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. 2023-04-16 00:56:20,863 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. started... 2023-04-16 00:56:20,863 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. as already flushing 2023-04-16 00:56:20,863 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. 2023-04-16 00:56:20,863 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:56:20,863 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:56:20,863 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:20,863 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:56:20,863 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:56:20,866 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:20,866 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:20,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:20,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:20,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:20,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:20,866 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:20,866 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:56:20,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:20,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:20,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:20,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:20,867 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:56:20,868 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@197bd581[Count = 0] remaining members to acquire global barrier 2023-04-16 00:56:20,868 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:56:20,868 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:20,869 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:20,869 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:20,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:20,869 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:56:20,869 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:56:20,869 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:56:20,869 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:20,869 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:56:20,870 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:56:20,870 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:56:20,870 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:56:20,870 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:20,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:20,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:20,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:20,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:20,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:20,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:20,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:20,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:20,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:20,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:20,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:56:20,873 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:56:20,873 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:56:20,873 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:56:20,873 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:56:20,873 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:56:20,873 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:56:20,874 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:20,874 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44548 deadline: 1681606640872, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:20,874 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:20,875 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:20,875 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:20,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:20,875 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:56:20,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:20,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:20,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:20,875 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:56:20,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:56:20,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:20,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:20,876 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:20,876 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44562 deadline: 1681606640873, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:20,876 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:20,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:20,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:20,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:20,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:20,877 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:20,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:20,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:20,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:20,881 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:20,881 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:20,881 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:20,881 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:56:20,881 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:20,881 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:56:20,881 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:56:20,881 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:56:20,881 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:20,881 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:56:20,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:56:20,881 DEBUG [Thread-4138] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:56:20,881 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:56:20,881 DEBUG [Thread-4138] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:56:20,881 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:20,881 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:20,881 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:56:20,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:56:20,969 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230416d487e7b08a26461c80379a6e28e04b68 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230416d487e7b08a26461c80379a6e28e04b68 2023-04-16 00:56:20,977 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=299, memsize=47.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp/A/6a2bcf616ac34837a1b83dcfc086d5fe 2023-04-16 00:56:20,981 DEBUG [Thread-4138] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:56:20,982 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:56:20,983 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:56:20,985 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:56:20,985 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:56:20,985 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:56:20,985 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:56:20,985 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:56:20,985 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:56:20,985 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:20,985 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:20,986 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:56:20,986 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:56:20,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:56:20,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:20,987 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:20,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:56:20,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:20,987 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:56:20,988 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:20,988 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:20,988 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:56:20,988 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:56:20,988 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:56:20,988 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:56:20,988 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:56:20,988 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:56:20,989 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:56:20,989 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:56:20,989 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. 2023-04-16 00:56:20,989 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. started... 2023-04-16 00:56:20,989 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. as already flushing 2023-04-16 00:56:20,989 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. 2023-04-16 00:56:20,989 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:56:20,989 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:56:20,989 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:20,989 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:56:20,989 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:56:20,991 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:20,991 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:20,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:20,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:20,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:20,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:20,992 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:20,992 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:56:20,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:20,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:20,993 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=299 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp/B/9cc616814cdc428a80348c74c031785b 2023-04-16 00:56:20,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:20,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:20,994 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:56:20,994 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7ddd054b[Count = 0] remaining members to acquire global barrier 2023-04-16 00:56:20,994 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:56:20,994 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:20,995 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:20,995 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:20,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:20,995 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:56:20,995 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:56:20,996 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:56:20,995 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:20,996 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:56:21,002 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:56:21,002 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:56:21,002 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:56:21,003 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:21,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:21,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:21,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:21,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:21,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:21,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:21,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:21,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:21,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:21,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:21,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:56:21,007 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:56:21,007 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:56:21,007 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:56:21,007 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:56:21,007 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:56:21,007 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:56:21,008 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=299 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp/C/10dcebe0daf24fecb73a2e5e4ea14010 2023-04-16 00:56:21,009 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:21,009 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:21,009 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:21,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:21,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:21,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:21,009 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:56:21,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:21,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:21,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:21,010 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:56:21,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:56:21,010 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:21,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:21,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:21,012 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp/A/6a2bcf616ac34837a1b83dcfc086d5fe as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A/6a2bcf616ac34837a1b83dcfc086d5fe 2023-04-16 00:56:21,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:21,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:21,013 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:21,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:21,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:21,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:21,016 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A/6a2bcf616ac34837a1b83dcfc086d5fe, entries=150, sequenceid=299, filesize=25.4 K 2023-04-16 00:56:21,016 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:56:21,016 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:56:21,016 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:56:21,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:56:21,016 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:56:21,016 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:56:21,016 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:56:21,016 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:21,016 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:21,016 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:21,016 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:21,017 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:21,017 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:21,017 DEBUG [Thread-4138] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:56:21,017 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp/B/9cc616814cdc428a80348c74c031785b as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/9cc616814cdc428a80348c74c031785b 2023-04-16 00:56:21,017 DEBUG [Thread-4138] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:56:21,017 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:56:21,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:56:21,017 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:56:21,017 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:21,020 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/9cc616814cdc428a80348c74c031785b, entries=150, sequenceid=299, filesize=11.9 K 2023-04-16 00:56:21,021 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp/C/10dcebe0daf24fecb73a2e5e4ea14010 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/10dcebe0daf24fecb73a2e5e4ea14010 2023-04-16 00:56:21,024 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/10dcebe0daf24fecb73a2e5e4ea14010, entries=150, sequenceid=299, filesize=11.9 K 2023-04-16 00:56:21,024 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~405.56 KB/415296, currentSize=40.25 KB/41220 for d538367277459b70f4a06235dfaacfec in 470ms, sequenceid=299, compaction requested=true 2023-04-16 00:56:21,024 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for d538367277459b70f4a06235dfaacfec: 2023-04-16 00:56:21,025 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 00:56:21,025 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 00:56:21,025 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-16 00:56:21,025 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-16 00:56:21,025 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 00:56:21,026 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 104278 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-16 00:56:21,026 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48872 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-16 00:56:21,026 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HStore(1912): d538367277459b70f4a06235dfaacfec/A is initiating minor compaction (all files) 2023-04-16 00:56:21,026 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HStore(1912): d538367277459b70f4a06235dfaacfec/B is initiating minor compaction (all files) 2023-04-16 00:56:21,026 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of d538367277459b70f4a06235dfaacfec/A in TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. 2023-04-16 00:56:21,026 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HRegion(2259): Starting compaction of d538367277459b70f4a06235dfaacfec/B in TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. 2023-04-16 00:56:21,027 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A/cd94a09e5f6e4122ac967486e097084c, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A/e5ac83fbb7684a0ca28b839dfff5231e, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A/cdd41307649a455fbb20b175cacaec3c, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A/6a2bcf616ac34837a1b83dcfc086d5fe] into tmpdir=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp, totalSize=101.8 K 2023-04-16 00:56:21,027 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/2c25715a76ec4aa9bbe7cdaa21f15cf5, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/98d55edf116f4da28ce9d4768d606e00, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/08ad153cb7df419da541f7ba065dd7e3, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/9cc616814cdc428a80348c74c031785b] into tmpdir=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp, totalSize=47.7 K 2023-04-16 00:56:21,027 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.Compactor(207): Compacting cd94a09e5f6e4122ac967486e097084c, keycount=150, bloomtype=ROW, size=25.7 K, encoding=NONE, compression=NONE, seqNum=212, earliestPutTs=1681606578421 2023-04-16 00:56:21,027 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting 2c25715a76ec4aa9bbe7cdaa21f15cf5, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=212, earliestPutTs=1681606578421 2023-04-16 00:56:21,027 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.Compactor(207): Compacting e5ac83fbb7684a0ca28b839dfff5231e, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=250, earliestPutTs=1681606578511 2023-04-16 00:56:21,027 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting 98d55edf116f4da28ce9d4768d606e00, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=250, earliestPutTs=1681606578511 2023-04-16 00:56:21,028 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.Compactor(207): Compacting cdd41307649a455fbb20b175cacaec3c, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=261, earliestPutTs=1681606579533 2023-04-16 00:56:21,028 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting 08ad153cb7df419da541f7ba065dd7e3, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=261, earliestPutTs=1681606579533 2023-04-16 00:56:21,028 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.Compactor(207): Compacting 6a2bcf616ac34837a1b83dcfc086d5fe, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=299, earliestPutTs=1681606580244 2023-04-16 00:56:21,028 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting 9cc616814cdc428a80348c74c031785b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=299, earliestPutTs=1681606580244 2023-04-16 00:56:21,036 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] throttle.PressureAwareThroughputController(145): d538367277459b70f4a06235dfaacfec#B#compaction#609 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-16 00:56:21,040 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] throttle.PressureAwareThroughputController(145): d538367277459b70f4a06235dfaacfec#A#compaction#608 average throughput is 4.93 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-16 00:56:21,057 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp/B/e45986fdbe5b4015af9ea0599530c7c4 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/e45986fdbe5b4015af9ea0599530c7c4 2023-04-16 00:56:21,062 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in d538367277459b70f4a06235dfaacfec/B of d538367277459b70f4a06235dfaacfec into e45986fdbe5b4015af9ea0599530c7c4(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 00:56:21,062 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for d538367277459b70f4a06235dfaacfec: 2023-04-16 00:56:21,062 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec., storeName=d538367277459b70f4a06235dfaacfec/B, priority=12, startTime=1681606581025; duration=0sec 2023-04-16 00:56:21,062 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 00:56:21,063 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-16 00:56:21,064 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48872 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-16 00:56:21,064 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HStore(1912): d538367277459b70f4a06235dfaacfec/C is initiating minor compaction (all files) 2023-04-16 00:56:21,064 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HRegion(2259): Starting compaction of d538367277459b70f4a06235dfaacfec/C in TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. 2023-04-16 00:56:21,064 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/0ab1f2532b1c4d8bbab20bddceec9d47, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/8546acaf4e1e47f0ba2fb98ced02e994, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/e645f6b6bf834a8fab0c6977593479d9, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/10dcebe0daf24fecb73a2e5e4ea14010] into tmpdir=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp, totalSize=47.7 K 2023-04-16 00:56:21,064 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting 0ab1f2532b1c4d8bbab20bddceec9d47, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=212, earliestPutTs=1681606578421 2023-04-16 00:56:21,065 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting 8546acaf4e1e47f0ba2fb98ced02e994, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=250, earliestPutTs=1681606578511 2023-04-16 00:56:21,065 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting e645f6b6bf834a8fab0c6977593479d9, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=261, earliestPutTs=1681606579533 2023-04-16 00:56:21,065 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting 10dcebe0daf24fecb73a2e5e4ea14010, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=299, earliestPutTs=1681606580244 2023-04-16 00:56:21,070 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] throttle.PressureAwareThroughputController(145): d538367277459b70f4a06235dfaacfec#C#compaction#610 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-16 00:56:21,079 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp/C/d27faa9987d741c48a2b7d699ff5c889 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/d27faa9987d741c48a2b7d699ff5c889 2023-04-16 00:56:21,082 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in d538367277459b70f4a06235dfaacfec/C of d538367277459b70f4a06235dfaacfec into d27faa9987d741c48a2b7d699ff5c889(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 00:56:21,082 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for d538367277459b70f4a06235dfaacfec: 2023-04-16 00:56:21,082 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec., storeName=d538367277459b70f4a06235dfaacfec/C, priority=12, startTime=1681606581025; duration=0sec 2023-04-16 00:56:21,082 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 00:56:21,117 DEBUG [Thread-4138] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:56:21,117 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:56:21,119 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:56:21,120 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:56:21,120 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:56:21,120 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:56:21,121 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:56:21,121 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:56:21,121 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:56:21,121 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:21,122 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:21,124 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:21,124 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:56:21,124 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:56:21,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:56:21,124 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:21,124 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:56:21,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:21,125 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:21,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:56:21,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:21,132 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:56:21,148 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:56:21,148 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:56:21,148 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:56:21,148 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:56:21,148 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:56:21,148 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:56:21,148 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:56:21,148 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. 2023-04-16 00:56:21,148 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. started... 2023-04-16 00:56:21,149 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing d538367277459b70f4a06235dfaacfec 3/3 column families, dataSize=40.25 KB heapSize=120.28 KB 2023-04-16 00:56:21,149 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d538367277459b70f4a06235dfaacfec, store=A 2023-04-16 00:56:21,149 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:56:21,149 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d538367277459b70f4a06235dfaacfec, store=B 2023-04-16 00:56:21,149 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:56:21,149 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d538367277459b70f4a06235dfaacfec, store=C 2023-04-16 00:56:21,149 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:56:21,178 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(9158): Flush requested on d538367277459b70f4a06235dfaacfec 2023-04-16 00:56:21,178 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. as already flushing 2023-04-16 00:56:21,180 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023041620f8e03ebc734e87ab5d28466379f224 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041620f8e03ebc734e87ab5d28466379f224 2023-04-16 00:56:21,205 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11854, totalHeapSize=27176, min timestamp=1681606581195, max timestamp=1681606581203Num uniques 859; 2023-04-16 00:56:21,205 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11854, totalHeapSize=27176, min timestamp=1681606581195, max timestamp=1681606581203Num uniques 859; 2023-04-16 00:56:21,205 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=11854, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-16 00:56:21,205 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11854, totalHeapSize=27176, min timestamp=1681606581195, max timestamp=1681606581203Num uniques 859; 2023-04-16 00:56:21,205 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=11854, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-16 00:56:21,205 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=11854, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-16 00:56:21,213 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=310, memsize=13.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp/A/74ed64aee6634e1db91e1b19e0f17a62 2023-04-16 00:56:21,227 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=310 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp/B/6140e76fab054aa4aa6ca6ceb9f73fba 2023-04-16 00:56:21,238 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:21,238 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44562 deadline: 1681606641234, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:21,240 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:21,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44548 deadline: 1681606641238, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:21,245 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=310 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp/C/03425eb8de8e46f78c8746cc7ef3baeb 2023-04-16 00:56:21,250 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp/A/74ed64aee6634e1db91e1b19e0f17a62 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A/74ed64aee6634e1db91e1b19e0f17a62 2023-04-16 00:56:21,254 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A/74ed64aee6634e1db91e1b19e0f17a62, entries=150, sequenceid=310, filesize=25.4 K 2023-04-16 00:56:21,255 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp/B/6140e76fab054aa4aa6ca6ceb9f73fba as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/6140e76fab054aa4aa6ca6ceb9f73fba 2023-04-16 00:56:21,258 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/6140e76fab054aa4aa6ca6ceb9f73fba, entries=150, sequenceid=310, filesize=11.9 K 2023-04-16 00:56:21,259 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp/C/03425eb8de8e46f78c8746cc7ef3baeb as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/03425eb8de8e46f78c8746cc7ef3baeb 2023-04-16 00:56:21,262 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/03425eb8de8e46f78c8746cc7ef3baeb, entries=150, sequenceid=310, filesize=11.9 K 2023-04-16 00:56:21,263 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~40.25 KB/41220, heapSize ~120.23 KB/123120, currentSize=140.89 KB/144270 for d538367277459b70f4a06235dfaacfec in 114ms, sequenceid=310, compaction requested=false 2023-04-16 00:56:21,263 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for d538367277459b70f4a06235dfaacfec: 2023-04-16 00:56:21,263 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. 2023-04-16 00:56:21,263 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:56:21,263 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:56:21,263 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:21,263 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:56:21,263 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:56:21,264 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:21,264 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:21,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:21,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:21,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:21,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:21,265 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:21,265 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:56:21,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:21,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:21,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:21,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:21,266 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:56:21,266 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3e75573d[Count = 0] remaining members to acquire global barrier 2023-04-16 00:56:21,266 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:56:21,266 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:21,267 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:21,267 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:21,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:21,268 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:56:21,268 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:21,268 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:56:21,268 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:56:21,268 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:56:21,270 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:21,270 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:56:21,270 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:56:21,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:21,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:21,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:21,270 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:56:21,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:21,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:21,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:21,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:21,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:21,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:21,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:21,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:56:21,272 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:56:21,272 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:56:21,272 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:56:21,272 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:56:21,273 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:56:21,273 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:56:21,279 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:21,279 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:56:21,279 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:21,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:21,280 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:21,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:21,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:21,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:21,280 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:56:21,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:56:21,280 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:21,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:21,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:21,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:21,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:21,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:21,281 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:21,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:21,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:21,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:21,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:21,284 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:21,284 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:21,284 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:21,284 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:56:21,284 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:56:21,284 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:56:21,284 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:56:21,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:56:21,284 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:56:21,284 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:56:21,284 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:21,284 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:21,285 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:21,285 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:21,285 DEBUG [Thread-4138] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:56:21,285 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:56:21,285 DEBUG [Thread-4138] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:56:21,285 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:56:21,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:56:21,342 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(9158): Flush requested on d538367277459b70f4a06235dfaacfec 2023-04-16 00:56:21,343 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing d538367277459b70f4a06235dfaacfec 3/3 column families, dataSize=147.60 KB heapSize=415.45 KB 2023-04-16 00:56:21,343 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d538367277459b70f4a06235dfaacfec, store=A 2023-04-16 00:56:21,343 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 00:56:21,343 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d538367277459b70f4a06235dfaacfec, store=B 2023-04-16 00:56:21,343 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 00:56:21,343 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d538367277459b70f4a06235dfaacfec, store=C 2023-04-16 00:56:21,343 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 00:56:21,353 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:21,353 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44562 deadline: 1681606641350, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:21,355 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:21,355 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44548 deadline: 1681606641353, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:21,385 DEBUG [Thread-4138] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:56:21,385 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:56:21,387 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:56:21,390 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:56:21,390 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:56:21,390 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:56:21,390 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:56:21,390 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:56:21,390 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:56:21,390 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:21,390 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:21,392 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:56:21,392 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:21,392 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:56:21,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:56:21,392 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:21,392 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:56:21,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:21,393 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:21,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:56:21,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:21,393 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:56:21,393 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:56:21,394 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:56:21,394 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:56:21,394 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:56:21,394 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:56:21,394 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:56:21,394 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. 2023-04-16 00:56:21,394 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. started... 2023-04-16 00:56:21,394 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:56:21,394 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. as already flushing 2023-04-16 00:56:21,394 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. 2023-04-16 00:56:21,394 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:56:21,394 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:56:21,394 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:21,394 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:56:21,394 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:56:21,396 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:21,397 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:21,397 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:56:21,397 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:21,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:21,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:21,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:21,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:21,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:21,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:21,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:21,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:21,398 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:56:21,399 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@68e4f07[Count = 0] remaining members to acquire global barrier 2023-04-16 00:56:21,399 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:56:21,399 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:21,401 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:21,401 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:21,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:21,401 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:56:21,401 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:56:21,401 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:21,401 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:56:21,401 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:56:21,403 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:21,403 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:56:21,403 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:56:21,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:21,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:21,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:21,403 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:56:21,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:21,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:21,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:21,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:21,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:21,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:21,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:21,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:56:21,406 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:56:21,406 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:56:21,406 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:56:21,406 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:56:21,406 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:56:21,406 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:56:21,408 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:21,408 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:56:21,408 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:21,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:21,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:21,408 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:21,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:21,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:21,409 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:56:21,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:56:21,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:21,409 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:21,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:21,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:21,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:21,410 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:21,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:21,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:21,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:21,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:21,413 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:21,413 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:21,413 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:21,413 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:21,413 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:21,413 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:56:21,413 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:56:21,413 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:21,413 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:56:21,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:56:21,413 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:56:21,413 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:56:21,413 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:21,413 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:56:21,413 DEBUG [Thread-4138] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:56:21,414 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:56:21,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:56:21,414 DEBUG [Thread-4138] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:56:21,414 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:56:21,455 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:21,455 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44562 deadline: 1681606641454, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:21,458 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:21,458 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44548 deadline: 1681606641456, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:21,469 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp/A/b2f263a74dc3430bbca4c42cdaccc326 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A/b2f263a74dc3430bbca4c42cdaccc326 2023-04-16 00:56:21,473 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in d538367277459b70f4a06235dfaacfec/A of d538367277459b70f4a06235dfaacfec into b2f263a74dc3430bbca4c42cdaccc326(size=26.0 K), total size for store is 51.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 00:56:21,473 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for d538367277459b70f4a06235dfaacfec: 2023-04-16 00:56:21,473 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec., storeName=d538367277459b70f4a06235dfaacfec/A, priority=12, startTime=1681606581025; duration=0sec 2023-04-16 00:56:21,473 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 00:56:21,514 DEBUG [Thread-4138] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:56:21,514 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:56:21,516 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:56:21,517 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:56:21,517 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:56:21,517 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:56:21,517 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:56:21,517 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:56:21,517 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:56:21,518 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:21,518 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:21,520 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:56:21,520 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:21,520 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:56:21,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:56:21,520 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:21,521 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:56:21,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:21,521 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:21,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:56:21,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:21,521 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:56:21,522 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:56:21,522 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:56:21,522 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:56:21,522 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:56:21,522 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:56:21,522 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:56:21,522 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:56:21,522 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. 2023-04-16 00:56:21,522 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. started... 2023-04-16 00:56:21,523 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. as already flushing 2023-04-16 00:56:21,523 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. 2023-04-16 00:56:21,523 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:56:21,523 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:56:21,523 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:21,523 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:56:21,523 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:56:21,524 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:21,524 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:21,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:21,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:21,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:21,525 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:21,525 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:56:21,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:21,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:21,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:21,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:21,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:21,526 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:56:21,526 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@411e458a[Count = 0] remaining members to acquire global barrier 2023-04-16 00:56:21,526 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:56:21,526 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:21,527 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:21,527 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:21,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:21,527 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:56:21,527 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:56:21,527 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:56:21,527 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:21,528 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:56:21,528 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:21,528 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:56:21,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:21,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:21,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:21,528 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:56:21,529 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:56:21,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:21,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:21,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:21,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:21,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:21,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:21,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:21,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:56:21,531 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:56:21,531 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:56:21,531 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:56:21,531 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:56:21,531 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:56:21,531 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:56:21,534 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:21,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:21,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:21,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:21,534 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:21,534 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:56:21,534 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:21,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:21,535 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:56:21,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:56:21,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:21,535 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:21,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:21,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:21,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:21,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:21,536 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:21,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:21,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:21,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:21,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:21,544 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:21,544 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:21,544 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:21,544 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:56:21,544 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:56:21,544 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:56:21,544 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:21,544 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:21,544 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:21,544 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:21,544 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:56:21,544 DEBUG [Thread-4138] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:56:21,544 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:56:21,544 DEBUG [Thread-4138] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:56:21,544 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:56:21,545 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:56:21,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:56:21,545 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:56:21,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:56:21,645 DEBUG [Thread-4138] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:56:21,645 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:56:21,647 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:56:21,648 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:56:21,648 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:56:21,648 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:56:21,648 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:56:21,648 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:56:21,648 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:56:21,648 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:21,648 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:21,650 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:21,650 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:56:21,650 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:56:21,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:56:21,650 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:21,650 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:56:21,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:21,650 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:21,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:56:21,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:21,651 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:56:21,651 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:56:21,651 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:56:21,651 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:56:21,651 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:56:21,651 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:56:21,651 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:56:21,651 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:56:21,651 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. 2023-04-16 00:56:21,651 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. started... 2023-04-16 00:56:21,652 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. as already flushing 2023-04-16 00:56:21,652 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. 2023-04-16 00:56:21,652 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:56:21,652 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:56:21,652 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:21,652 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:56:21,652 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:56:21,655 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:21,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:21,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:21,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:21,655 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:21,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:21,655 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:21,655 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:56:21,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:21,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:21,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:21,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:21,657 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:56:21,657 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@547ce7bf[Count = 0] remaining members to acquire global barrier 2023-04-16 00:56:21,657 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:56:21,657 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:21,658 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:21,658 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:21,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:21,658 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:56:21,658 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:56:21,658 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:56:21,658 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:21,658 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:56:21,659 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:21,659 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44562 deadline: 1681606641657, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:21,659 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:21,659 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:56:21,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:21,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:21,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:21,659 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:56:21,660 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:56:21,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:21,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:21,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:21,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:21,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:21,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:21,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:21,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:56:21,662 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:21,662 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:56:21,662 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:56:21,662 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:56:21,662 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:56:21,662 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:56:21,662 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:56:21,662 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44548 deadline: 1681606641659, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:21,664 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:21,664 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:21,664 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:21,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:21,664 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:56:21,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:21,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:21,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:21,665 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:56:21,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:56:21,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:21,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:21,665 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:21,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:21,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:21,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:21,666 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:21,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:21,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:21,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:21,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:21,674 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:21,674 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:21,674 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:56:21,674 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:56:21,674 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:56:21,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:56:21,674 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:21,674 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:21,674 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:21,674 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:21,674 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:21,674 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:56:21,674 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:56:21,674 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:56:21,674 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:56:21,674 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:56:21,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:56:21,674 DEBUG [Thread-4138] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:56:21,674 DEBUG [Thread-4138] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:56:21,758 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230416fcb0cbeb4953451f939d0815734b422f to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230416fcb0cbeb4953451f939d0815734b422f 2023-04-16 00:56:21,775 DEBUG [Thread-4138] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:56:21,775 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:56:21,777 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:56:21,778 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:56:21,778 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:56:21,778 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:56:21,778 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:56:21,778 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:56:21,778 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:56:21,779 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:21,779 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:21,780 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:56:21,780 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:56:21,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:56:21,780 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:21,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:21,781 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:21,781 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:56:21,781 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:21,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:56:21,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:21,781 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:56:21,782 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:56:21,782 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:56:21,782 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:56:21,782 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:56:21,782 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:56:21,782 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:56:21,782 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:56:21,782 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. 2023-04-16 00:56:21,782 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. started... 2023-04-16 00:56:21,782 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. as already flushing 2023-04-16 00:56:21,782 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. 2023-04-16 00:56:21,782 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:56:21,782 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:56:21,782 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:21,782 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:56:21,783 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:56:21,784 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:21,784 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:21,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:21,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:21,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:21,784 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:21,784 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:56:21,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:21,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:21,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:21,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:21,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:21,786 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:56:21,786 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@40248eb2[Count = 0] remaining members to acquire global barrier 2023-04-16 00:56:21,786 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:56:21,786 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:21,788 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:21,788 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:56:21,788 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:21,788 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:21,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:21,788 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:56:21,788 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:56:21,788 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:56:21,791 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:21,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:21,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:21,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:21,791 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:56:21,791 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:56:21,791 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:56:21,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:21,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:21,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:21,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:21,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:21,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:21,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:21,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:56:21,794 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:56:21,794 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:56:21,794 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:56:21,794 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:56:21,794 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:56:21,794 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:56:21,797 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:21,797 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:56:21,797 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:21,797 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:21,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:21,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:21,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:21,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:21,797 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:56:21,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:56:21,797 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:21,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:21,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:21,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:21,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:21,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:21,799 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:21,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:21,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:21,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:21,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:21,802 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:21,802 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:21,802 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:21,802 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:56:21,802 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:56:21,802 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:21,802 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:21,802 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:21,802 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:21,802 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:56:21,802 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:56:21,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:56:21,802 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:56:21,802 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:56:21,802 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:56:21,802 DEBUG [Thread-4138] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:56:21,802 DEBUG [Thread-4138] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:56:21,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:56:21,903 DEBUG [Thread-4138] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:56:21,903 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:56:21,904 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:56:21,906 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:56:21,906 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:56:21,906 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:56:21,906 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:56:21,906 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:56:21,906 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:56:21,906 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:21,906 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:21,908 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:21,908 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:56:21,908 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:56:21,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:56:21,908 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:21,908 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:56:21,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:21,909 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:21,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:56:21,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:21,909 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:56:21,909 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:56:21,909 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:56:21,909 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:56:21,909 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:56:21,909 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:56:21,910 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:56:21,910 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. 2023-04-16 00:56:21,910 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. started... 2023-04-16 00:56:21,910 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:56:21,910 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. as already flushing 2023-04-16 00:56:21,910 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. 2023-04-16 00:56:21,910 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:56:21,910 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:56:21,910 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:21,910 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:56:21,910 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:56:21,912 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:21,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:21,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:21,913 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:21,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:21,913 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:21,913 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:56:21,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:21,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:21,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:21,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:21,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:21,916 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:56:21,916 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@34a607c6[Count = 0] remaining members to acquire global barrier 2023-04-16 00:56:21,916 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:56:21,916 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:21,917 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:21,917 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:21,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:21,917 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:56:21,917 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:56:21,917 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:56:21,917 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:21,918 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:56:21,919 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:21,919 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:56:21,919 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:56:21,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:21,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:21,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:21,919 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:56:21,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:21,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:21,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:21,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:21,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:21,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:21,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:21,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:56:21,921 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:56:21,921 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:56:21,922 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:56:21,922 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:56:21,922 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:56:21,922 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:56:21,923 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:21,923 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:56:21,923 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:21,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:21,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:21,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:21,923 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:21,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:21,924 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:56:21,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:21,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:56:21,924 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:21,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:21,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:21,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:21,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:21,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:21,925 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:21,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:21,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:21,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:21,931 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:21,932 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:56:21,932 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:21,932 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:21,932 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:56:21,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:56:21,932 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:56:21,932 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:56:21,932 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:56:21,932 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:56:21,932 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:56:21,932 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:21,932 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:21,932 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:21,932 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:21,932 DEBUG [Thread-4138] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:56:21,932 DEBUG [Thread-4138] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:56:21,932 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:56:21,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:56:21,963 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:21,963 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44562 deadline: 1681606641961, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:21,965 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:21,965 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44548 deadline: 1681606641963, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:22,033 DEBUG [Thread-4138] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:56:22,033 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:56:22,035 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:56:22,035 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:56:22,036 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:56:22,036 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:56:22,036 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:56:22,036 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:56:22,036 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:56:22,036 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:22,036 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:22,037 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:56:22,037 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:22,037 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:56:22,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:56:22,038 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:22,038 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:56:22,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:22,038 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:22,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:56:22,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:22,039 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:56:22,039 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:56:22,039 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:56:22,039 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:56:22,039 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:56:22,039 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:56:22,039 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:56:22,039 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. 2023-04-16 00:56:22,039 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. started... 2023-04-16 00:56:22,039 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. as already flushing 2023-04-16 00:56:22,040 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. 2023-04-16 00:56:22,040 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:56:22,040 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:56:22,040 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:22,040 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:56:22,040 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:56:22,041 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:22,041 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:22,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:22,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:22,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:22,041 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:22,042 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:56:22,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:22,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:22,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:22,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:22,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:22,043 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:56:22,043 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@72726cd[Count = 0] remaining members to acquire global barrier 2023-04-16 00:56:22,043 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:56:22,043 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:22,044 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:22,044 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:22,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:22,044 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:56:22,044 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:56:22,044 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:22,044 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:56:22,044 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:56:22,046 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:56:22,046 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:22,046 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:56:22,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:22,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:22,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:22,046 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:56:22,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:22,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:22,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:22,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:22,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:22,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:22,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:22,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:56:22,049 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:56:22,049 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:56:22,049 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:56:22,049 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:56:22,049 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:56:22,049 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:56:22,050 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:22,050 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:22,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:22,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:22,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:22,050 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:56:22,050 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:22,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:22,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:22,051 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:56:22,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:56:22,051 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:22,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:22,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:22,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:22,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:22,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:22,052 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:22,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:22,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:22,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:22,054 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:56:22,054 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:56:22,054 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:22,054 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:22,054 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:56:22,054 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:56:22,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:56:22,054 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:56:22,055 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:56:22,054 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:22,055 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:22,055 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:22,055 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:22,055 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:22,055 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:56:22,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:56:22,055 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:56:22,055 DEBUG [Thread-4138] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:56:22,055 DEBUG [Thread-4138] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:56:22,155 DEBUG [Thread-4138] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:56:22,156 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:56:22,158 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:56:22,159 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:56:22,159 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:56:22,159 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:56:22,159 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:56:22,159 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:56:22,159 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:56:22,159 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:22,159 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:22,160 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:56:22,160 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:22,160 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:56:22,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:56:22,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:22,161 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:22,161 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:56:22,161 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:22,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:56:22,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:22,162 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:56:22,162 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:56:22,162 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:56:22,162 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:56:22,162 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:56:22,162 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:56:22,162 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:56:22,162 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:56:22,162 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. 2023-04-16 00:56:22,162 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. started... 2023-04-16 00:56:22,162 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. as already flushing 2023-04-16 00:56:22,162 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. 2023-04-16 00:56:22,163 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:56:22,163 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:56:22,163 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:22,163 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:56:22,163 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:56:22,165 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:22,165 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:22,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:22,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:22,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:22,165 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:22,165 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:56:22,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:22,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:22,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:22,166 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=347, memsize=49.2 K, hasBloomFilter=true, into tmp file hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp/A/fb09760c8d434f05b086841c634310f8 2023-04-16 00:56:22,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:22,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:22,168 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:56:22,169 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@34556889[Count = 0] remaining members to acquire global barrier 2023-04-16 00:56:22,169 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:56:22,169 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:22,170 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:22,170 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:22,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:22,170 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:56:22,170 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:56:22,170 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:56:22,171 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:22,171 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:56:22,172 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:22,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:22,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:22,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:22,172 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:56:22,172 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:56:22,172 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:56:22,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:22,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:22,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:22,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:22,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:22,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:22,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:22,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:56:22,176 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:56:22,176 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:56:22,176 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:56:22,176 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:56:22,176 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:56:22,176 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:56:22,178 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:22,178 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:56:22,178 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:22,178 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:22,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:22,179 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=347 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp/B/e5965a0411c44477b420d28bd13dd872 2023-04-16 00:56:22,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:22,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:22,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:22,179 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:56:22,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:56:22,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:22,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:22,180 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:22,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:22,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:22,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:22,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:22,183 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:22,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:22,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:22,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:22,188 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:22,188 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:22,188 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:56:22,188 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:56:22,188 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:22,188 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:56:22,188 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:56:22,188 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:22,188 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:22,188 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:22,188 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:22,188 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:56:22,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:56:22,188 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:56:22,188 DEBUG [Thread-4138] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:56:22,189 DEBUG [Thread-4138] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:56:22,189 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=347 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp/C/058046d40cea48548d46e33010220d87 2023-04-16 00:56:22,189 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:56:22,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:56:22,193 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp/A/fb09760c8d434f05b086841c634310f8 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A/fb09760c8d434f05b086841c634310f8 2023-04-16 00:56:22,197 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A/fb09760c8d434f05b086841c634310f8, entries=200, sequenceid=347, filesize=32.3 K 2023-04-16 00:56:22,198 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp/B/e5965a0411c44477b420d28bd13dd872 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/e5965a0411c44477b420d28bd13dd872 2023-04-16 00:56:22,201 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/e5965a0411c44477b420d28bd13dd872, entries=150, sequenceid=347, filesize=11.9 K 2023-04-16 00:56:22,202 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp/C/058046d40cea48548d46e33010220d87 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/058046d40cea48548d46e33010220d87 2023-04-16 00:56:22,205 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/058046d40cea48548d46e33010220d87, entries=150, sequenceid=347, filesize=11.9 K 2023-04-16 00:56:22,205 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~147.60 KB/151140, heapSize ~415.41 KB/425376, currentSize=33.54 KB/34350 for d538367277459b70f4a06235dfaacfec in 863ms, sequenceid=347, compaction requested=true 2023-04-16 00:56:22,205 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for d538367277459b70f4a06235dfaacfec: 2023-04-16 00:56:22,205 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 00:56:22,205 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 00:56:22,205 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-16 00:56:22,205 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-16 00:56:22,205 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 00:56:22,206 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37172 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-16 00:56:22,206 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 85758 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-16 00:56:22,206 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HStore(1912): d538367277459b70f4a06235dfaacfec/B is initiating minor compaction (all files) 2023-04-16 00:56:22,206 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HStore(1912): d538367277459b70f4a06235dfaacfec/A is initiating minor compaction (all files) 2023-04-16 00:56:22,206 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HRegion(2259): Starting compaction of d538367277459b70f4a06235dfaacfec/B in TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. 2023-04-16 00:56:22,206 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of d538367277459b70f4a06235dfaacfec/A in TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. 2023-04-16 00:56:22,206 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/e45986fdbe5b4015af9ea0599530c7c4, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/6140e76fab054aa4aa6ca6ceb9f73fba, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/e5965a0411c44477b420d28bd13dd872] into tmpdir=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp, totalSize=36.3 K 2023-04-16 00:56:22,206 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A/b2f263a74dc3430bbca4c42cdaccc326, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A/74ed64aee6634e1db91e1b19e0f17a62, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A/fb09760c8d434f05b086841c634310f8] into tmpdir=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp, totalSize=83.7 K 2023-04-16 00:56:22,207 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting e45986fdbe5b4015af9ea0599530c7c4, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=299, earliestPutTs=1681606580244 2023-04-16 00:56:22,208 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting 6140e76fab054aa4aa6ca6ceb9f73fba, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=310, earliestPutTs=1681606580560 2023-04-16 00:56:22,208 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.Compactor(207): Compacting b2f263a74dc3430bbca4c42cdaccc326, keycount=150, bloomtype=ROW, size=26.0 K, encoding=NONE, compression=NONE, seqNum=299, earliestPutTs=1681606580244 2023-04-16 00:56:22,208 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting e5965a0411c44477b420d28bd13dd872, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=347, earliestPutTs=1681606581223 2023-04-16 00:56:22,208 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.Compactor(207): Compacting 74ed64aee6634e1db91e1b19e0f17a62, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=310, earliestPutTs=1681606580560 2023-04-16 00:56:22,209 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.Compactor(207): Compacting fb09760c8d434f05b086841c634310f8, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=347, earliestPutTs=1681606581223 2023-04-16 00:56:22,214 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] throttle.PressureAwareThroughputController(145): d538367277459b70f4a06235dfaacfec#B#compaction#617 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-16 00:56:22,225 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] throttle.PressureAwareThroughputController(145): d538367277459b70f4a06235dfaacfec#A#compaction#618 average throughput is 1.97 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-16 00:56:22,227 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp/B/000d29b1df9e4c9f8b4fd8726653f465 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/000d29b1df9e4c9f8b4fd8726653f465 2023-04-16 00:56:22,231 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in d538367277459b70f4a06235dfaacfec/B of d538367277459b70f4a06235dfaacfec into 000d29b1df9e4c9f8b4fd8726653f465(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 00:56:22,231 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for d538367277459b70f4a06235dfaacfec: 2023-04-16 00:56:22,231 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec., storeName=d538367277459b70f4a06235dfaacfec/B, priority=13, startTime=1681606582205; duration=0sec 2023-04-16 00:56:22,231 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 00:56:22,231 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-16 00:56:22,232 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37172 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-16 00:56:22,232 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HStore(1912): d538367277459b70f4a06235dfaacfec/C is initiating minor compaction (all files) 2023-04-16 00:56:22,232 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HRegion(2259): Starting compaction of d538367277459b70f4a06235dfaacfec/C in TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. 2023-04-16 00:56:22,232 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/d27faa9987d741c48a2b7d699ff5c889, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/03425eb8de8e46f78c8746cc7ef3baeb, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/058046d40cea48548d46e33010220d87] into tmpdir=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp, totalSize=36.3 K 2023-04-16 00:56:22,232 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting d27faa9987d741c48a2b7d699ff5c889, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=299, earliestPutTs=1681606580244 2023-04-16 00:56:22,233 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting 03425eb8de8e46f78c8746cc7ef3baeb, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=310, earliestPutTs=1681606580560 2023-04-16 00:56:22,233 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting 058046d40cea48548d46e33010220d87, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=347, earliestPutTs=1681606581223 2023-04-16 00:56:22,240 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] throttle.PressureAwareThroughputController(145): d538367277459b70f4a06235dfaacfec#C#compaction#619 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-16 00:56:22,269 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp/A/fe26a1dd9f1a47328a6b6173cc96d91b as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A/fe26a1dd9f1a47328a6b6173cc96d91b 2023-04-16 00:56:22,273 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in d538367277459b70f4a06235dfaacfec/A of d538367277459b70f4a06235dfaacfec into fe26a1dd9f1a47328a6b6173cc96d91b(size=26.1 K), total size for store is 26.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 00:56:22,273 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for d538367277459b70f4a06235dfaacfec: 2023-04-16 00:56:22,273 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec., storeName=d538367277459b70f4a06235dfaacfec/A, priority=13, startTime=1681606582205; duration=0sec 2023-04-16 00:56:22,273 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 00:56:22,289 DEBUG [Thread-4138] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:56:22,290 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:56:22,292 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:56:22,293 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:56:22,293 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:56:22,293 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:56:22,293 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:56:22,293 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:56:22,293 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:56:22,293 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:22,293 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:22,297 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:56:22,297 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:56:22,297 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:22,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:56:22,298 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:22,298 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:56:22,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:22,298 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:22,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:56:22,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:22,298 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:56:22,299 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:56:22,299 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:56:22,299 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:56:22,299 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:56:22,299 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:56:22,299 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:56:22,300 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:56:22,300 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. 2023-04-16 00:56:22,300 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. started... 2023-04-16 00:56:22,300 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing d538367277459b70f4a06235dfaacfec 3/3 column families, dataSize=33.54 KB heapSize=100.36 KB 2023-04-16 00:56:22,300 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d538367277459b70f4a06235dfaacfec, store=A 2023-04-16 00:56:22,300 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:56:22,300 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d538367277459b70f4a06235dfaacfec, store=B 2023-04-16 00:56:22,300 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:56:22,300 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d538367277459b70f4a06235dfaacfec, store=C 2023-04-16 00:56:22,300 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:56:22,313 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230416823e82c393b74b8b80b927619c6b0c85 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230416823e82c393b74b8b80b927619c6b0c85 2023-04-16 00:56:22,318 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=358, memsize=11.2 K, hasBloomFilter=true, into tmp file hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp/A/da12a06b31d441d599d829b0ff669323 2023-04-16 00:56:22,329 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=358 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp/B/f62174af8c2047a3b8ad29743a771078 2023-04-16 00:56:22,346 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=358 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp/C/26bc504829a9431ba5e28f9fd59417f0 2023-04-16 00:56:22,351 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp/A/da12a06b31d441d599d829b0ff669323 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A/da12a06b31d441d599d829b0ff669323 2023-04-16 00:56:22,356 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A/da12a06b31d441d599d829b0ff669323, entries=100, sequenceid=358, filesize=18.6 K 2023-04-16 00:56:22,357 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp/B/f62174af8c2047a3b8ad29743a771078 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/f62174af8c2047a3b8ad29743a771078 2023-04-16 00:56:22,360 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/f62174af8c2047a3b8ad29743a771078, entries=100, sequenceid=358, filesize=9.5 K 2023-04-16 00:56:22,360 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp/C/26bc504829a9431ba5e28f9fd59417f0 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/26bc504829a9431ba5e28f9fd59417f0 2023-04-16 00:56:22,364 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/26bc504829a9431ba5e28f9fd59417f0, entries=100, sequenceid=358, filesize=9.5 K 2023-04-16 00:56:22,364 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~100.31 KB/102720, currentSize=0 B/0 for d538367277459b70f4a06235dfaacfec in 64ms, sequenceid=358, compaction requested=false 2023-04-16 00:56:22,364 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for d538367277459b70f4a06235dfaacfec: 2023-04-16 00:56:22,364 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. 2023-04-16 00:56:22,364 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:56:22,365 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:56:22,365 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:22,365 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:56:22,365 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:56:22,366 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:22,366 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:22,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:22,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:22,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:22,366 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:22,366 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:56:22,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:22,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:22,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:22,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:22,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:22,369 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:56:22,369 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@506031af[Count = 0] remaining members to acquire global barrier 2023-04-16 00:56:22,369 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:56:22,369 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:22,370 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:22,370 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:22,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:22,370 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:56:22,370 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:56:22,370 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:56:22,370 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:22,370 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:56:22,372 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:22,372 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:56:22,372 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:56:22,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:22,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:22,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:22,372 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:56:22,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:22,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:22,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:22,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:22,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:22,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:22,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:22,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:56:22,375 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:56:22,375 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:56:22,375 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:56:22,375 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:56:22,375 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:56:22,375 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:56:22,376 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:22,376 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:22,376 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:22,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:22,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:22,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:22,376 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:56:22,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:22,377 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:56:22,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:22,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:56:22,377 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:22,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:22,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:22,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:22,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:22,378 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:22,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:22,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:22,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:22,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:22,380 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:56:22,380 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:56:22,380 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:56:22,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:56:22,380 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:22,380 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:56:22,380 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:56:22,380 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:56:22,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:56:22,380 DEBUG [Thread-4138] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:56:22,380 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:56:22,381 DEBUG [Thread-4138] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:56:22,380 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:56:22,380 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:22,381 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:22,381 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:22,381 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:22,381 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:22,381 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:22,481 DEBUG [Thread-4138] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:56:22,481 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:56:22,483 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:56:22,484 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:56:22,484 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:56:22,484 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:56:22,484 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:56:22,484 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:56:22,484 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:56:22,485 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:22,485 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:22,489 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:56:22,489 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:22,489 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:56:22,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:56:22,490 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:22,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:22,490 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:56:22,490 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:22,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:56:22,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:22,490 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:56:22,491 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:56:22,491 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:56:22,491 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:56:22,491 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:56:22,491 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:56:22,491 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:56:22,491 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. 2023-04-16 00:56:22,491 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:56:22,491 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. started... 2023-04-16 00:56:22,491 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing d538367277459b70f4a06235dfaacfec 3/3 column families, dataSize=20.13 KB heapSize=60.52 KB 2023-04-16 00:56:22,492 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d538367277459b70f4a06235dfaacfec, store=A 2023-04-16 00:56:22,492 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:56:22,492 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d538367277459b70f4a06235dfaacfec, store=B 2023-04-16 00:56:22,492 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:56:22,492 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d538367277459b70f4a06235dfaacfec, store=C 2023-04-16 00:56:22,492 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:56:22,504 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023041696706b8065ca4ccfaf28b59ff03952b5 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041696706b8065ca4ccfaf28b59ff03952b5 2023-04-16 00:56:22,522 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. as already flushing 2023-04-16 00:56:22,523 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(9158): Flush requested on d538367277459b70f4a06235dfaacfec 2023-04-16 00:56:22,584 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681606582576, max timestamp=1681606582582Num uniques 859; 2023-04-16 00:56:22,584 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-16 00:56:22,584 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681606582576, max timestamp=1681606582582Num uniques 859; 2023-04-16 00:56:22,585 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-16 00:56:22,585 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681606582576, max timestamp=1681606582582Num uniques 859; 2023-04-16 00:56:22,585 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-16 00:56:22,613 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1681606582604, max timestamp=1681606582611Num uniques 1018; 2023-04-16 00:56:22,613 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-04-16 00:56:22,614 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1681606582604, max timestamp=1681606582611Num uniques 1018; 2023-04-16 00:56:22,614 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-04-16 00:56:22,614 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1681606582604, max timestamp=1681606582611Num uniques 1018; 2023-04-16 00:56:22,615 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-04-16 00:56:22,644 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8095, totalHeapSize=18648, min timestamp=1681606582627, max timestamp=1681606582641Num uniques 1077; 2023-04-16 00:56:22,644 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8098, totalHeapSize=18648, min timestamp=1681606582627, max timestamp=1681606582641Num uniques 1077; 2023-04-16 00:56:22,644 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=49315, new segment data size=8095, suffix heap size=139976,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=177 2023-04-16 00:56:22,644 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8097, totalHeapSize=18648, min timestamp=1681606582627, max timestamp=1681606582641Num uniques 1077; 2023-04-16 00:56:22,645 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=49317, new segment data size=8097, suffix heap size=139976,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=177 2023-04-16 00:56:22,644 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=49318, new segment data size=8098, suffix heap size=139976,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=177 2023-04-16 00:56:22,663 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp/C/8ab0b6d870784a6b9aa7ee9be8159c36 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/8ab0b6d870784a6b9aa7ee9be8159c36 2023-04-16 00:56:22,667 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in d538367277459b70f4a06235dfaacfec/C of d538367277459b70f4a06235dfaacfec into 8ab0b6d870784a6b9aa7ee9be8159c36(size=12.6 K), total size for store is 22.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 00:56:22,667 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for d538367277459b70f4a06235dfaacfec: 2023-04-16 00:56:22,667 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec., storeName=d538367277459b70f4a06235dfaacfec/C, priority=13, startTime=1681606582205; duration=0sec 2023-04-16 00:56:22,667 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 00:56:22,673 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=336, cellSize=15386, totalHeapSize=35184, min timestamp=1681606582660, max timestamp=1681606582671Num uniques 1036; 2023-04-16 00:56:22,673 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=15386, suffix heap size=135712,new segment heap size=35184 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=336 2023-04-16 00:56:22,674 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=336, cellSize=15386, totalHeapSize=35184, min timestamp=1681606582660, max timestamp=1681606582671Num uniques 1036; 2023-04-16 00:56:22,674 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=15386, suffix heap size=135712,new segment heap size=35184 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=336 2023-04-16 00:56:22,674 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=336, cellSize=15386, totalHeapSize=35184, min timestamp=1681606582660, max timestamp=1681606582671Num uniques 1036; 2023-04-16 00:56:22,674 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=15386, suffix heap size=135712,new segment heap size=35184 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=336 2023-04-16 00:56:22,689 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=245, cellSize=11220, totalHeapSize=25720, min timestamp=1681606582682, max timestamp=1681606582687Num uniques 1195; 2023-04-16 00:56:22,689 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=54730, new segment data size=11220, suffix heap size=152248,new segment heap size=25720 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1195, new segment cells count=245 2023-04-16 00:56:22,689 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=245, cellSize=11220, totalHeapSize=25720, min timestamp=1681606582682, max timestamp=1681606582687Num uniques 1195; 2023-04-16 00:56:22,690 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=54730, new segment data size=11220, suffix heap size=152248,new segment heap size=25720 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1195, new segment cells count=245 2023-04-16 00:56:22,690 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=245, cellSize=11220, totalHeapSize=25720, min timestamp=1681606582682, max timestamp=1681606582687Num uniques 1195; 2023-04-16 00:56:22,690 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=54730, new segment data size=11220, suffix heap size=152248,new segment heap size=25720 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1195, new segment cells count=245 2023-04-16 00:56:22,712 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:22,712 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 182 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44548 deadline: 1681606642709, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:22,712 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:22,712 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:44564 deadline: 1681606642709, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:22,713 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:22,713 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44562 deadline: 1681606642711, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:22,714 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=204, cellSize=9340, totalHeapSize=21456, min timestamp=1681606582689, max timestamp=1681606582710Num uniques 1104; 2023-04-16 00:56:22,714 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=50560, new segment data size=9340, suffix heap size=142784,new segment heap size=21456 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1104, new segment cells count=204 2023-04-16 00:56:22,714 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=204, cellSize=9340, totalHeapSize=21456, min timestamp=1681606582689, max timestamp=1681606582710Num uniques 1104; 2023-04-16 00:56:22,714 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=204, cellSize=9340, totalHeapSize=21456, min timestamp=1681606582689, max timestamp=1681606582710Num uniques 1104; 2023-04-16 00:56:22,714 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=50560, new segment data size=9340, suffix heap size=142784,new segment heap size=21456 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1104, new segment cells count=204 2023-04-16 00:56:22,714 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=50560, new segment data size=9340, suffix heap size=142784,new segment heap size=21456 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1104, new segment cells count=204 2023-04-16 00:56:22,741 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=163, cellSize=7458, totalHeapSize=17192, min timestamp=1681606582725, max timestamp=1681606582738Num uniques 1063; 2023-04-16 00:56:22,741 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=163, cellSize=7458, totalHeapSize=17192, min timestamp=1681606582725, max timestamp=1681606582738Num uniques 1063; 2023-04-16 00:56:22,741 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=48678, new segment data size=7458, suffix heap size=138520,new segment heap size=17192 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1063, new segment cells count=163 2023-04-16 00:56:22,741 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=163, cellSize=7458, totalHeapSize=17192, min timestamp=1681606582725, max timestamp=1681606582738Num uniques 1063; 2023-04-16 00:56:22,742 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=48678, new segment data size=7458, suffix heap size=138520,new segment heap size=17192 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1063, new segment cells count=163 2023-04-16 00:56:22,741 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=48678, new segment data size=7458, suffix heap size=138520,new segment heap size=17192 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1063, new segment cells count=163 2023-04-16 00:56:22,771 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=172, cellSize=7872, totalHeapSize=18128, min timestamp=1681606582757, max timestamp=1681606582769Num uniques 1022; 2023-04-16 00:56:22,771 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=46802, new segment data size=7872, suffix heap size=134256,new segment heap size=18128 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1022, new segment cells count=172 2023-04-16 00:56:22,771 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=172, cellSize=7872, totalHeapSize=18128, min timestamp=1681606582757, max timestamp=1681606582769Num uniques 1022; 2023-04-16 00:56:22,771 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=46802, new segment data size=7872, suffix heap size=134256,new segment heap size=18128 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1022, new segment cells count=172 2023-04-16 00:56:22,771 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=172, cellSize=7872, totalHeapSize=18128, min timestamp=1681606582757, max timestamp=1681606582769Num uniques 1022; 2023-04-16 00:56:22,771 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=46802, new segment data size=7872, suffix heap size=134256,new segment heap size=18128 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1022, new segment cells count=172 2023-04-16 00:56:22,795 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=181, cellSize=8286, totalHeapSize=19064, min timestamp=1681606582786, max timestamp=1681606582792Num uniques 1031; 2023-04-16 00:56:22,795 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=47216, new segment data size=8286, suffix heap size=135192,new segment heap size=19064 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1031, new segment cells count=181 2023-04-16 00:56:22,795 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=181, cellSize=8286, totalHeapSize=19064, min timestamp=1681606582786, max timestamp=1681606582792Num uniques 1031; 2023-04-16 00:56:22,795 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=181, cellSize=8286, totalHeapSize=19064, min timestamp=1681606582786, max timestamp=1681606582792Num uniques 1031; 2023-04-16 00:56:22,795 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=47216, new segment data size=8286, suffix heap size=135192,new segment heap size=19064 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1031, new segment cells count=181 2023-04-16 00:56:22,795 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=47216, new segment data size=8286, suffix heap size=135192,new segment heap size=19064 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1031, new segment cells count=181 2023-04-16 00:56:22,821 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=240, cellSize=10990, totalHeapSize=25200, min timestamp=1681606582811, max timestamp=1681606582819Num uniques 1040; 2023-04-16 00:56:22,821 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=47630, new segment data size=10990, suffix heap size=136128,new segment heap size=25200 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1040, new segment cells count=240 2023-04-16 00:56:22,822 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=240, cellSize=10990, totalHeapSize=25200, min timestamp=1681606582811, max timestamp=1681606582819Num uniques 1040; 2023-04-16 00:56:22,822 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=47630, new segment data size=10990, suffix heap size=136128,new segment heap size=25200 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1040, new segment cells count=240 2023-04-16 00:56:22,822 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=240, cellSize=10990, totalHeapSize=25200, min timestamp=1681606582811, max timestamp=1681606582819Num uniques 1040; 2023-04-16 00:56:22,822 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=47630, new segment data size=10990, suffix heap size=136128,new segment heap size=25200 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1040, new segment cells count=240 2023-04-16 00:56:22,838 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=249, cellSize=11404, totalHeapSize=26136, min timestamp=1681606582826, max timestamp=1681606582835Num uniques 1099; 2023-04-16 00:56:22,838 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=50334, new segment data size=11404, suffix heap size=142264,new segment heap size=26136 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1099, new segment cells count=249 2023-04-16 00:56:22,838 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=249, cellSize=11404, totalHeapSize=26136, min timestamp=1681606582826, max timestamp=1681606582835Num uniques 1099; 2023-04-16 00:56:22,838 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=50334, new segment data size=11404, suffix heap size=142264,new segment heap size=26136 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1099, new segment cells count=249 2023-04-16 00:56:22,838 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=249, cellSize=11404, totalHeapSize=26136, min timestamp=1681606582826, max timestamp=1681606582835Num uniques 1099; 2023-04-16 00:56:22,838 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=50334, new segment data size=11404, suffix heap size=142264,new segment heap size=26136 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1099, new segment cells count=249 2023-04-16 00:56:22,854 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=190, cellSize=8700, totalHeapSize=20000, min timestamp=1681606582846, max timestamp=1681606582852Num uniques 1108; 2023-04-16 00:56:22,854 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=50738, new segment data size=8700, suffix heap size=143200,new segment heap size=20000 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1108, new segment cells count=190 2023-04-16 00:56:22,855 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=208, cellSize=9508, totalHeapSize=21872, min timestamp=1681606582846, max timestamp=1681606582852Num uniques 1108; 2023-04-16 00:56:22,855 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=208, cellSize=9518, totalHeapSize=21872, min timestamp=1681606582846, max timestamp=1681606582852Num uniques 1108; 2023-04-16 00:56:22,855 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=50738, new segment data size=9518, suffix heap size=143200,new segment heap size=21872 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1108, new segment cells count=208 2023-04-16 00:56:22,855 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=50728, new segment data size=9508, suffix heap size=143200,new segment heap size=21872 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1108, new segment cells count=208 2023-04-16 00:56:22,872 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=267, cellSize=12222, totalHeapSize=28008, min timestamp=1681606582862, max timestamp=1681606582869Num uniques 1067; 2023-04-16 00:56:22,872 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=267, cellSize=12222, totalHeapSize=28008, min timestamp=1681606582862, max timestamp=1681606582869Num uniques 1067; 2023-04-16 00:56:22,872 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=48862, new segment data size=12222, suffix heap size=138936,new segment heap size=28008 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1067, new segment cells count=267 2023-04-16 00:56:22,872 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=267, cellSize=12222, totalHeapSize=28008, min timestamp=1681606582862, max timestamp=1681606582869Num uniques 1049; 2023-04-16 00:56:22,872 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=48044, new segment data size=12222, suffix heap size=137064,new segment heap size=28008 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1049, new segment cells count=267 2023-04-16 00:56:22,872 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=48862, new segment data size=12222, suffix heap size=138936,new segment heap size=28008 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1067, new segment cells count=267 2023-04-16 00:56:22,909 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:22,909 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:22,909 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 190 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44562 deadline: 1681606642905, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:22,909 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44556 deadline: 1681606642905, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:22,909 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:22,909 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44554 deadline: 1681606642905, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:22,910 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=176, cellSize=8046, totalHeapSize=18544, min timestamp=1681606582894, max timestamp=1681606582908Num uniques 1126; 2023-04-16 00:56:22,910 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=176, cellSize=8046, totalHeapSize=18544, min timestamp=1681606582894, max timestamp=1681606582908Num uniques 1126; 2023-04-16 00:56:22,911 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=51556, new segment data size=8046, suffix heap size=145072,new segment heap size=18544 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1126, new segment cells count=176 2023-04-16 00:56:22,911 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=176, cellSize=8051, totalHeapSize=18544, min timestamp=1681606582894, max timestamp=1681606582908Num uniques 1126; 2023-04-16 00:56:22,911 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=51556, new segment data size=8046, suffix heap size=145072,new segment heap size=18544 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1126, new segment cells count=176 2023-04-16 00:56:22,911 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=51561, new segment data size=8051, suffix heap size=145072,new segment heap size=18544 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1126, new segment cells count=176 2023-04-16 00:56:22,922 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=364, memsize=6.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp/A/22f90baaf5e44bbd86a35ea3cd725c23 2023-04-16 00:56:22,947 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=185, cellSize=8460, totalHeapSize=19480, min timestamp=1681606582934, max timestamp=1681606582944Num uniques 1035; 2023-04-16 00:56:22,947 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=47390, new segment data size=8460, suffix heap size=135608,new segment heap size=19480 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1035, new segment cells count=185 2023-04-16 00:56:22,947 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=185, cellSize=8466, totalHeapSize=19480, min timestamp=1681606582934, max timestamp=1681606582944Num uniques 1035; 2023-04-16 00:56:22,947 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=47396, new segment data size=8466, suffix heap size=135608,new segment heap size=19480 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1035, new segment cells count=185 2023-04-16 00:56:22,947 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=185, cellSize=8465, totalHeapSize=19480, min timestamp=1681606582934, max timestamp=1681606582944Num uniques 1035; 2023-04-16 00:56:22,947 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=47395, new segment data size=8465, suffix heap size=135608,new segment heap size=19480 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1035, new segment cells count=185 2023-04-16 00:56:22,976 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=244, cellSize=11174, totalHeapSize=25616, min timestamp=1681606582962, max timestamp=1681606582974Num uniques 1044; 2023-04-16 00:56:22,976 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=47814, new segment data size=11174, suffix heap size=136544,new segment heap size=25616 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1044, new segment cells count=244 2023-04-16 00:56:22,977 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=244, cellSize=11174, totalHeapSize=25616, min timestamp=1681606582962, max timestamp=1681606582974Num uniques 1044; 2023-04-16 00:56:22,977 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=47814, new segment data size=11174, suffix heap size=136544,new segment heap size=25616 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1044, new segment cells count=244 2023-04-16 00:56:22,977 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=244, cellSize=11174, totalHeapSize=25616, min timestamp=1681606582962, max timestamp=1681606582974Num uniques 1044; 2023-04-16 00:56:22,977 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=47814, new segment data size=11174, suffix heap size=136544,new segment heap size=25616 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1044, new segment cells count=244 2023-04-16 00:56:23,009 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=153, cellSize=7005, totalHeapSize=16152, min timestamp=1681606583001, max timestamp=1681606583007Num uniques 1103; 2023-04-16 00:56:23,009 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=50515, new segment data size=7005, suffix heap size=142680,new segment heap size=16152 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1103, new segment cells count=153 2023-04-16 00:56:23,010 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=153, cellSize=7005, totalHeapSize=16152, min timestamp=1681606583001, max timestamp=1681606583007Num uniques 1103; 2023-04-16 00:56:23,010 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=153, cellSize=7005, totalHeapSize=16152, min timestamp=1681606583001, max timestamp=1681606583007Num uniques 1103; 2023-04-16 00:56:23,010 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=50515, new segment data size=7005, suffix heap size=142680,new segment heap size=16152 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1103, new segment cells count=153 2023-04-16 00:56:23,010 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=50515, new segment data size=7005, suffix heap size=142680,new segment heap size=16152 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1103, new segment cells count=153 2023-04-16 00:56:23,025 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=262, cellSize=11992, totalHeapSize=27488, min timestamp=1681606583016, max timestamp=1681606583023Num uniques 1012; 2023-04-16 00:56:23,025 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=46342, new segment data size=11992, suffix heap size=133216,new segment heap size=27488 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1012, new segment cells count=262 2023-04-16 00:56:23,025 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=262, cellSize=11992, totalHeapSize=27488, min timestamp=1681606583016, max timestamp=1681606583023Num uniques 1012; 2023-04-16 00:56:23,025 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=46342, new segment data size=11992, suffix heap size=133216,new segment heap size=27488 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1012, new segment cells count=262 2023-04-16 00:56:23,026 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1681606583016, max timestamp=1681606583023Num uniques 1012; 2023-04-16 00:56:23,026 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=46342, new segment data size=6870, suffix heap size=133216,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1012, new segment cells count=150 2023-04-16 00:56:23,043 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=271, cellSize=12406, totalHeapSize=28424, min timestamp=1681606583036, max timestamp=1681606583040Num uniques 1121; 2023-04-16 00:56:23,043 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=271, cellSize=12406, totalHeapSize=28424, min timestamp=1681606583036, max timestamp=1681606583040Num uniques 1121; 2023-04-16 00:56:23,043 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=51336, new segment data size=12406, suffix heap size=144552,new segment heap size=28424 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1121, new segment cells count=271 2023-04-16 00:56:23,043 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=271, cellSize=12406, totalHeapSize=28424, min timestamp=1681606583036, max timestamp=1681606583040Num uniques 1009; 2023-04-16 00:56:23,043 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=51336, new segment data size=12406, suffix heap size=144552,new segment heap size=28424 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1121, new segment cells count=271 2023-04-16 00:56:23,043 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=46214, new segment data size=12406, suffix heap size=132904,new segment heap size=28424 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=271 2023-04-16 00:56:23,059 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=180, cellSize=8240, totalHeapSize=18960, min timestamp=1681606583052, max timestamp=1681606583056Num uniques 1130; 2023-04-16 00:56:23,059 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=51750, new segment data size=8240, suffix heap size=145488,new segment heap size=18960 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1130, new segment cells count=180 2023-04-16 00:56:23,059 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=180, cellSize=8240, totalHeapSize=18960, min timestamp=1681606583052, max timestamp=1681606583056Num uniques 1130; 2023-04-16 00:56:23,059 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=180, cellSize=8240, totalHeapSize=18960, min timestamp=1681606583052, max timestamp=1681606583056Num uniques 1130; 2023-04-16 00:56:23,059 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=51750, new segment data size=8240, suffix heap size=145488,new segment heap size=18960 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1130, new segment cells count=180 2023-04-16 00:56:23,059 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=51750, new segment data size=8240, suffix heap size=145488,new segment heap size=18960 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1130, new segment cells count=180 2023-04-16 00:56:23,074 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=189, cellSize=8654, totalHeapSize=19896, min timestamp=1681606583063, max timestamp=1681606583072Num uniques 1039; 2023-04-16 00:56:23,074 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=47584, new segment data size=8654, suffix heap size=136024,new segment heap size=19896 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1039, new segment cells count=189 2023-04-16 00:56:23,075 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=189, cellSize=8654, totalHeapSize=19896, min timestamp=1681606583063, max timestamp=1681606583072Num uniques 1039; 2023-04-16 00:56:23,075 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=47584, new segment data size=8654, suffix heap size=136024,new segment heap size=19896 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1039, new segment cells count=189 2023-04-16 00:56:23,075 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=189, cellSize=8654, totalHeapSize=19896, min timestamp=1681606583063, max timestamp=1681606583072Num uniques 1039; 2023-04-16 00:56:23,075 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=47584, new segment data size=8654, suffix heap size=136024,new segment heap size=19896 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1039, new segment cells count=189 2023-04-16 00:56:23,093 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=298, cellSize=13648, totalHeapSize=31232, min timestamp=1681606583084, max timestamp=1681606583090Num uniques 1048; 2023-04-16 00:56:23,093 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=47998, new segment data size=13648, suffix heap size=136960,new segment heap size=31232 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1048, new segment cells count=298 2023-04-16 00:56:23,094 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=298, cellSize=13648, totalHeapSize=31232, min timestamp=1681606583084, max timestamp=1681606583090Num uniques 1048; 2023-04-16 00:56:23,094 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=298, cellSize=13646, totalHeapSize=31232, min timestamp=1681606583084, max timestamp=1681606583091Num uniques 1048; 2023-04-16 00:56:23,094 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=47998, new segment data size=13648, suffix heap size=136960,new segment heap size=31232 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1048, new segment cells count=298 2023-04-16 00:56:23,094 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=47996, new segment data size=13646, suffix heap size=136960,new segment heap size=31232 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1048, new segment cells count=298 2023-04-16 00:56:23,110 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:23,110 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 213 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44562 deadline: 1681606643105, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:23,110 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=157, cellSize=7185, totalHeapSize=16568, min timestamp=1681606583100, max timestamp=1681606583108Num uniques 1157; 2023-04-16 00:56:23,110 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=157, cellSize=7185, totalHeapSize=16568, min timestamp=1681606583100, max timestamp=1681606583108Num uniques 1157; 2023-04-16 00:56:23,110 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:23,111 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=52985, new segment data size=7185, suffix heap size=148296,new segment heap size=16568 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1157, new segment cells count=157 2023-04-16 00:56:23,111 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 179 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44554 deadline: 1681606643106, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:23,110 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=52985, new segment data size=7185, suffix heap size=148296,new segment heap size=16568 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1157, new segment cells count=157 2023-04-16 00:56:23,110 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=157, cellSize=7185, totalHeapSize=16568, min timestamp=1681606583100, max timestamp=1681606583108Num uniques 1157; 2023-04-16 00:56:23,111 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=52985, new segment data size=7185, suffix heap size=148296,new segment heap size=16568 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1157, new segment cells count=157 2023-04-16 00:56:23,133 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=166, cellSize=7590, totalHeapSize=17504, min timestamp=1681606583124, max timestamp=1681606583131Num uniques 1016; 2023-04-16 00:56:23,133 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=46520, new segment data size=7590, suffix heap size=133632,new segment heap size=17504 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1016, new segment cells count=166 2023-04-16 00:56:23,134 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=166, cellSize=7591, totalHeapSize=17504, min timestamp=1681606583124, max timestamp=1681606583131Num uniques 1016; 2023-04-16 00:56:23,134 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=46521, new segment data size=7591, suffix heap size=133632,new segment heap size=17504 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1016, new segment cells count=166 2023-04-16 00:56:23,134 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=166, cellSize=7596, totalHeapSize=17504, min timestamp=1681606583124, max timestamp=1681606583131Num uniques 1016; 2023-04-16 00:56:23,134 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=46526, new segment data size=7596, suffix heap size=133632,new segment heap size=17504 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1016, new segment cells count=166 2023-04-16 00:56:23,154 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=225, cellSize=10300, totalHeapSize=23640, min timestamp=1681606583147, max timestamp=1681606583152Num uniques 1025; 2023-04-16 00:56:23,154 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=46940, new segment data size=10300, suffix heap size=134568,new segment heap size=23640 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1025, new segment cells count=225 2023-04-16 00:56:23,155 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=225, cellSize=10300, totalHeapSize=23640, min timestamp=1681606583147, max timestamp=1681606583152Num uniques 1025; 2023-04-16 00:56:23,155 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=46940, new segment data size=10300, suffix heap size=134568,new segment heap size=23640 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1025, new segment cells count=225 2023-04-16 00:56:23,155 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=225, cellSize=10300, totalHeapSize=23640, min timestamp=1681606583147, max timestamp=1681606583152Num uniques 1025; 2023-04-16 00:56:23,156 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=46940, new segment data size=10300, suffix heap size=134568,new segment heap size=23640 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1025, new segment cells count=225 2023-04-16 00:56:23,176 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=184, cellSize=8424, totalHeapSize=19376, min timestamp=1681606583170, max timestamp=1681606583174Num uniques 1084; 2023-04-16 00:56:23,176 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=184, cellSize=8424, totalHeapSize=19376, min timestamp=1681606583170, max timestamp=1681606583174Num uniques 1084; 2023-04-16 00:56:23,176 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1681606583170, max timestamp=1681606583174Num uniques 1084; 2023-04-16 00:56:23,176 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=49644, new segment data size=6870, suffix heap size=140704,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1084, new segment cells count=150 2023-04-16 00:56:23,176 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=49644, new segment data size=8424, suffix heap size=140704,new segment heap size=19376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1084, new segment cells count=184 2023-04-16 00:56:23,176 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=49644, new segment data size=8424, suffix heap size=140704,new segment heap size=19376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1084, new segment cells count=184 2023-04-16 00:56:23,201 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=243, cellSize=11118, totalHeapSize=25512, min timestamp=1681606583191, max timestamp=1681606583199Num uniques 1043; 2023-04-16 00:56:23,201 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=47758, new segment data size=11118, suffix heap size=136440,new segment heap size=25512 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1043, new segment cells count=243 2023-04-16 00:56:23,202 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=243, cellSize=11123, totalHeapSize=25512, min timestamp=1681606583191, max timestamp=1681606583199Num uniques 1043; 2023-04-16 00:56:23,202 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=47763, new segment data size=11123, suffix heap size=136440,new segment heap size=25512 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1043, new segment cells count=243 2023-04-16 00:56:23,202 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=243, cellSize=11121, totalHeapSize=25512, min timestamp=1681606583191, max timestamp=1681606583199Num uniques 1009; 2023-04-16 00:56:23,202 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=46207, new segment data size=11121, suffix heap size=132904,new segment heap size=25512 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=243 2023-04-16 00:56:23,220 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=202, cellSize=9250, totalHeapSize=21248, min timestamp=1681606583210, max timestamp=1681606583217Num uniques 1102; 2023-04-16 00:56:23,220 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=202, cellSize=9250, totalHeapSize=21248, min timestamp=1681606583210, max timestamp=1681606583217Num uniques 1102; 2023-04-16 00:56:23,220 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=50470, new segment data size=9250, suffix heap size=142576,new segment heap size=21248 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1102, new segment cells count=202 2023-04-16 00:56:23,220 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=202, cellSize=9250, totalHeapSize=21248, min timestamp=1681606583210, max timestamp=1681606583217Num uniques 1102; 2023-04-16 00:56:23,220 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=50470, new segment data size=9250, suffix heap size=142576,new segment heap size=21248 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1102, new segment cells count=202 2023-04-16 00:56:23,220 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=50470, new segment data size=9250, suffix heap size=142576,new segment heap size=21248 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1102, new segment cells count=202 2023-04-16 00:56:23,235 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=211, cellSize=9656, totalHeapSize=22184, min timestamp=1681606583231, max timestamp=1681606583234Num uniques 1061; 2023-04-16 00:56:23,236 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=48586, new segment data size=9656, suffix heap size=138312,new segment heap size=22184 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1061, new segment cells count=211 2023-04-16 00:56:23,236 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=211, cellSize=9656, totalHeapSize=22184, min timestamp=1681606583231, max timestamp=1681606583234Num uniques 1061; 2023-04-16 00:56:23,237 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=211, cellSize=9656, totalHeapSize=22184, min timestamp=1681606583231, max timestamp=1681606583234Num uniques 1061; 2023-04-16 00:56:23,237 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=48586, new segment data size=9656, suffix heap size=138312,new segment heap size=22184 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1061, new segment cells count=211 2023-04-16 00:56:23,237 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=48586, new segment data size=9656, suffix heap size=138312,new segment heap size=22184 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1061, new segment cells count=211 2023-04-16 00:56:23,305 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:23,305 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 281 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44548 deadline: 1681606643282, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:23,306 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=170, cellSize=7772, totalHeapSize=17920, min timestamp=1681606583278, max timestamp=1681606583285Num uniques 1070; 2023-04-16 00:56:23,306 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=48992, new segment data size=7772, suffix heap size=139248,new segment heap size=17920 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1070, new segment cells count=170 2023-04-16 00:56:23,306 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=170, cellSize=7776, totalHeapSize=17920, min timestamp=1681606583278, max timestamp=1681606583285Num uniques 1070; 2023-04-16 00:56:23,306 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=170, cellSize=7770, totalHeapSize=17920, min timestamp=1681606583278, max timestamp=1681606583285Num uniques 1070; 2023-04-16 00:56:23,307 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=48996, new segment data size=7776, suffix heap size=139248,new segment heap size=17920 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1070, new segment cells count=170 2023-04-16 00:56:23,307 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=48990, new segment data size=7770, suffix heap size=139248,new segment heap size=17920 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1070, new segment cells count=170 2023-04-16 00:56:23,324 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=229, cellSize=10484, totalHeapSize=24056, min timestamp=1681606583310, max timestamp=1681606583321Num uniques 1029; 2023-04-16 00:56:23,324 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=47124, new segment data size=10484, suffix heap size=134984,new segment heap size=24056 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1029, new segment cells count=229 2023-04-16 00:56:23,324 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=229, cellSize=10484, totalHeapSize=24056, min timestamp=1681606583310, max timestamp=1681606583321Num uniques 1029; 2023-04-16 00:56:23,324 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=47124, new segment data size=10484, suffix heap size=134984,new segment heap size=24056 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1029, new segment cells count=229 2023-04-16 00:56:23,326 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=179, cellSize=8194, totalHeapSize=18856, min timestamp=1681606583310, max timestamp=1681606583321Num uniques 1029; 2023-04-16 00:56:23,326 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=47124, new segment data size=8194, suffix heap size=134984,new segment heap size=18856 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1029, new segment cells count=179 2023-04-16 00:56:23,339 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=364 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp/B/0454d7b6b16e44c79e571242d0b3a390 2023-04-16 00:56:23,342 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=238, cellSize=10898, totalHeapSize=24992, min timestamp=1681606583331, max timestamp=1681606583340Num uniques 1088; 2023-04-16 00:56:23,342 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=49828, new segment data size=10898, suffix heap size=141120,new segment heap size=24992 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1088, new segment cells count=238 2023-04-16 00:56:23,343 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=238, cellSize=10898, totalHeapSize=24992, min timestamp=1681606583331, max timestamp=1681606583340Num uniques 1038; 2023-04-16 00:56:23,343 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=47538, new segment data size=10898, suffix heap size=135920,new segment heap size=24992 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1038, new segment cells count=238 2023-04-16 00:56:23,343 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=238, cellSize=10898, totalHeapSize=24992, min timestamp=1681606583331, max timestamp=1681606583340Num uniques 1088; 2023-04-16 00:56:23,343 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=49828, new segment data size=10898, suffix heap size=141120,new segment heap size=24992 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1088, new segment cells count=238 2023-04-16 00:56:23,353 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=364 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp/C/fa97bea0af4e4f5491453e35a38bd085 2023-04-16 00:56:23,358 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp/A/22f90baaf5e44bbd86a35ea3cd725c23 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A/22f90baaf5e44bbd86a35ea3cd725c23 2023-04-16 00:56:23,360 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=247, cellSize=11312, totalHeapSize=25928, min timestamp=1681606583340, max timestamp=1681606583358Num uniques 1097; 2023-04-16 00:56:23,360 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=50242, new segment data size=11312, suffix heap size=142056,new segment heap size=25928 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1097, new segment cells count=247 2023-04-16 00:56:23,361 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=247, cellSize=11312, totalHeapSize=25928, min timestamp=1681606583340, max timestamp=1681606583358Num uniques 1097; 2023-04-16 00:56:23,361 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=50242, new segment data size=11312, suffix heap size=142056,new segment heap size=25928 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1097, new segment cells count=247 2023-04-16 00:56:23,361 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=247, cellSize=11312, totalHeapSize=25928, min timestamp=1681606583340, max timestamp=1681606583358Num uniques 1097; 2023-04-16 00:56:23,361 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=50242, new segment data size=11312, suffix heap size=142056,new segment heap size=25928 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1097, new segment cells count=247 2023-04-16 00:56:23,364 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A/22f90baaf5e44bbd86a35ea3cd725c23, entries=150, sequenceid=364, filesize=25.4 K 2023-04-16 00:56:23,365 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp/B/0454d7b6b16e44c79e571242d0b3a390 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/0454d7b6b16e44c79e571242d0b3a390 2023-04-16 00:56:23,369 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/0454d7b6b16e44c79e571242d0b3a390, entries=50, sequenceid=364, filesize=7.1 K 2023-04-16 00:56:23,369 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp/C/fa97bea0af4e4f5491453e35a38bd085 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/fa97bea0af4e4f5491453e35a38bd085 2023-04-16 00:56:23,374 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/fa97bea0af4e4f5491453e35a38bd085, entries=50, sequenceid=364, filesize=7.1 K 2023-04-16 00:56:23,375 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~20.13 KB/20610, heapSize ~60.47 KB/61920, currentSize=114.05 KB/116790 for d538367277459b70f4a06235dfaacfec in 883ms, sequenceid=364, compaction requested=true 2023-04-16 00:56:23,375 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for d538367277459b70f4a06235dfaacfec: 2023-04-16 00:56:23,375 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. 2023-04-16 00:56:23,375 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:56:23,375 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:56:23,375 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:23,375 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:56:23,375 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:56:23,377 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:23,377 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:23,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:23,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:23,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:23,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:23,377 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:23,377 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:56:23,377 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(9158): Flush requested on d538367277459b70f4a06235dfaacfec 2023-04-16 00:56:23,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:23,377 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing d538367277459b70f4a06235dfaacfec 3/3 column families, dataSize=120.76 KB heapSize=336.89 KB 2023-04-16 00:56:23,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:23,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:23,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:23,378 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:56:23,378 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@54fc8452[Count = 0] remaining members to acquire global barrier 2023-04-16 00:56:23,378 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:56:23,379 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:23,379 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d538367277459b70f4a06235dfaacfec, store=A 2023-04-16 00:56:23,379 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 00:56:23,379 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d538367277459b70f4a06235dfaacfec, store=B 2023-04-16 00:56:23,379 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 00:56:23,379 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d538367277459b70f4a06235dfaacfec, store=C 2023-04-16 00:56:23,379 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 00:56:23,380 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:23,380 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:23,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:23,380 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:56:23,380 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:56:23,380 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:56:23,381 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:23,381 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:56:23,382 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:23,382 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:56:23,382 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:56:23,382 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:56:23,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:23,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:23,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:23,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:23,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:23,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:23,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:23,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:23,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:23,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:23,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:56:23,386 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:56:23,386 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:56:23,386 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:56:23,386 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:56:23,386 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:56:23,386 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:56:23,388 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:23,388 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 206 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44554 deadline: 1681606643385, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:23,388 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:23,388 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 202 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44564 deadline: 1681606643386, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:23,389 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:23,389 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:23,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:23,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:23,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:23,389 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:23,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:23,389 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:56:23,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:23,390 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:56:23,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:56:23,390 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:23,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:23,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:23,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:23,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:23,391 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:23,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:23,391 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:23,391 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 222 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44556 deadline: 1681606643388, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:23,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:23,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:23,392 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:23,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:23,392 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 243 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44562 deadline: 1681606643389, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:23,393 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:56:23,393 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:56:23,393 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:56:23,394 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:56:23,393 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:56:23,393 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:56:23,393 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:23,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:56:23,394 DEBUG [Thread-4138] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:56:23,394 DEBUG [Thread-4138] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:56:23,394 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:56:23,394 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:23,394 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:23,394 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:23,394 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:23,394 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:23,394 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:23,394 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:56:23,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:56:23,397 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304160821f03004e74223a4b2cddc235af7d5 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304160821f03004e74223a4b2cddc235af7d5 2023-04-16 00:56:23,401 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=949, memsize=42.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp/A/7582b7a5f2454c368c0ad8a4637a6015 2023-04-16 00:56:23,409 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:23,409 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 283 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44548 deadline: 1681606643407, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:23,490 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:23,490 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 208 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44554 deadline: 1681606643489, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:23,491 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:23,491 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 204 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44564 deadline: 1681606643489, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:23,494 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:23,494 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 224 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44556 deadline: 1681606643492, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:23,494 DEBUG [Thread-4138] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:56:23,495 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:56:23,495 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:23,495 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 245 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44562 deadline: 1681606643494, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:23,496 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:56:23,497 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:56:23,497 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:56:23,497 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:56:23,498 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:56:23,498 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:56:23,498 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:56:23,498 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:23,498 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:23,499 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:56:23,499 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:23,499 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:56:23,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:56:23,499 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:23,500 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:56:23,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:23,500 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:23,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:56:23,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:23,500 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:56:23,501 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:56:23,501 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:56:23,501 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:56:23,501 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:56:23,501 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:56:23,501 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:56:23,501 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:56:23,501 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. 2023-04-16 00:56:23,501 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. started... 2023-04-16 00:56:23,501 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. as already flushing 2023-04-16 00:56:23,501 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. 2023-04-16 00:56:23,501 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:56:23,501 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:56:23,501 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:23,501 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:56:23,502 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:56:23,503 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:23,503 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:23,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:23,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:23,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:23,503 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:23,503 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:56:23,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:23,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:23,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:23,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:23,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:23,505 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:56:23,505 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6a53c14e[Count = 0] remaining members to acquire global barrier 2023-04-16 00:56:23,505 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:56:23,505 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:23,508 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:23,508 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:23,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:23,508 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:56:23,508 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:56:23,508 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:56:23,508 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:23,508 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:56:23,509 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:23,509 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:56:23,509 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:56:23,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:23,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:23,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:23,509 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:56:23,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:23,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:23,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:23,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:23,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:23,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:23,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:23,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:56:23,512 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:56:23,512 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:56:23,512 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:56:23,512 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:56:23,512 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:56:23,512 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:56:23,514 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:23,514 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:23,514 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:23,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:23,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:23,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:23,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:23,514 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:56:23,514 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:56:23,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:56:23,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:23,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:23,514 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:23,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:23,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:23,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:23,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:23,515 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:23,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:23,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:23,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:23,517 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:23,517 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:56:23,517 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:23,518 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:56:23,518 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:56:23,518 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:56:23,518 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:56:23,518 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:56:23,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:56:23,518 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:23,518 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:23,518 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:23,518 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:23,518 DEBUG [Thread-4138] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:56:23,518 DEBUG [Thread-4138] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:56:23,518 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:23,518 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:56:23,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:56:23,618 DEBUG [Thread-4138] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:56:23,618 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:23,619 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 285 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44548 deadline: 1681606643616, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:23,619 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:56:23,622 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:56:23,623 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:56:23,623 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:56:23,623 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:56:23,623 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:56:23,623 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:56:23,623 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:56:23,623 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:23,624 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:23,626 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:23,626 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:56:23,626 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:56:23,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:56:23,626 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:23,626 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:56:23,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:23,626 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:23,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:56:23,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:23,627 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:56:23,627 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:56:23,627 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:56:23,627 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:56:23,627 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:56:23,627 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:56:23,627 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. 2023-04-16 00:56:23,627 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. started... 2023-04-16 00:56:23,627 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:56:23,627 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. as already flushing 2023-04-16 00:56:23,627 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. 2023-04-16 00:56:23,627 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:56:23,627 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:56:23,627 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:23,627 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:56:23,627 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:56:23,629 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:23,629 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:23,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:23,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:23,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:23,629 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:23,629 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:56:23,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:23,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:23,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:23,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:23,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:23,630 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:56:23,630 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@755dfdad[Count = 0] remaining members to acquire global barrier 2023-04-16 00:56:23,630 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:56:23,630 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:23,631 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:23,631 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:23,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:23,631 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:56:23,632 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:56:23,632 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:23,632 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:56:23,632 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:56:23,633 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:23,633 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:56:23,633 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:56:23,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:23,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:23,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:23,633 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:56:23,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:23,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:23,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:23,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:23,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:23,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:23,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:23,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:56:23,635 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:56:23,635 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:56:23,635 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:56:23,635 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:56:23,635 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:56:23,635 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:56:23,638 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:23,638 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:56:23,638 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:23,638 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:23,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:23,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:23,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:23,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:23,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:23,639 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:56:23,639 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:23,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:56:23,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:23,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:23,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:23,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:23,640 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:23,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:23,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:23,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:23,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:23,642 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:23,642 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:23,642 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:23,642 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:56:23,642 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:56:23,642 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:56:23,642 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:23,642 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:56:23,642 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:23,642 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:56:23,642 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:23,642 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:56:23,643 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:23,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:56:23,642 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:56:23,643 DEBUG [Thread-4138] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:56:23,643 DEBUG [Thread-4138] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:56:23,643 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:56:23,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:56:23,693 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:23,693 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 210 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44554 deadline: 1681606643691, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:23,695 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:23,695 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 206 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44564 deadline: 1681606643693, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:23,699 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:23,699 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 247 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44562 deadline: 1681606643696, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:23,699 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:23,699 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 226 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44556 deadline: 1681606643696, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:23,743 DEBUG [Thread-4138] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:56:23,744 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:56:23,745 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:56:23,746 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:56:23,746 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:56:23,746 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:56:23,746 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:56:23,747 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:56:23,747 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:56:23,747 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:23,747 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:23,748 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:56:23,748 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:23,748 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:56:23,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:56:23,749 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:23,749 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:56:23,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:23,749 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:23,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:56:23,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:23,750 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:56:23,750 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:56:23,750 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:56:23,750 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:56:23,750 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:56:23,750 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:56:23,750 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:56:23,750 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:56:23,750 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. 2023-04-16 00:56:23,750 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. started... 2023-04-16 00:56:23,751 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. as already flushing 2023-04-16 00:56:23,751 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. 2023-04-16 00:56:23,751 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:56:23,751 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:56:23,751 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:23,751 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:56:23,751 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:56:23,752 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:23,752 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:23,753 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:23,753 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:56:23,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:23,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:23,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:23,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:23,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:23,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:23,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:23,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:23,755 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:56:23,755 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@246157e0[Count = 0] remaining members to acquire global barrier 2023-04-16 00:56:23,755 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:56:23,755 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:23,759 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:23,759 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:23,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:23,760 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:56:23,760 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:56:23,760 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:23,760 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:56:23,760 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:56:23,762 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:56:23,762 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:23,762 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:56:23,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:23,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:23,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:23,762 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:56:23,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:23,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:23,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:23,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:23,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:23,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:23,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:23,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:56:23,765 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:56:23,765 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:56:23,765 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:56:23,765 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:56:23,765 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:56:23,765 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:56:23,766 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:23,766 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:23,766 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:23,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:23,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:23,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:23,766 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:56:23,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:23,767 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:56:23,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:56:23,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:23,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:23,767 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:23,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:23,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:23,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:23,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:23,768 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:23,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:23,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:23,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:23,770 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:23,770 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:56:23,770 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:23,771 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:56:23,771 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:56:23,771 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:56:23,771 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:56:23,771 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:56:23,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:56:23,771 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:23,770 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:56:23,771 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:23,771 DEBUG [Thread-4138] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:56:23,771 DEBUG [Thread-4138] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:56:23,771 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:23,771 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:23,771 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:56:23,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:56:23,771 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:23,811 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=949 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp/B/98f28f3e808b4fdabf1fa12b5c18e302 2023-04-16 00:56:23,821 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=949 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp/C/58df185bf1f14e7ebda08f4cc70d58d4 2023-04-16 00:56:23,825 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp/A/7582b7a5f2454c368c0ad8a4637a6015 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A/7582b7a5f2454c368c0ad8a4637a6015 2023-04-16 00:56:23,829 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A/7582b7a5f2454c368c0ad8a4637a6015, entries=250, sequenceid=949, filesize=39.2 K 2023-04-16 00:56:23,830 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp/B/98f28f3e808b4fdabf1fa12b5c18e302 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/98f28f3e808b4fdabf1fa12b5c18e302 2023-04-16 00:56:23,833 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/98f28f3e808b4fdabf1fa12b5c18e302, entries=150, sequenceid=949, filesize=11.9 K 2023-04-16 00:56:23,834 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp/C/58df185bf1f14e7ebda08f4cc70d58d4 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/58df185bf1f14e7ebda08f4cc70d58d4 2023-04-16 00:56:23,837 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/58df185bf1f14e7ebda08f4cc70d58d4, entries=150, sequenceid=949, filesize=11.9 K 2023-04-16 00:56:23,837 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~356.77 KB/365328, currentSize=60.38 KB/61830 for d538367277459b70f4a06235dfaacfec in 460ms, sequenceid=949, compaction requested=true 2023-04-16 00:56:23,837 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for d538367277459b70f4a06235dfaacfec: 2023-04-16 00:56:23,838 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 00:56:23,838 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-16 00:56:23,838 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=3), splitQueue=0 2023-04-16 00:56:23,838 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-16 00:56:23,838 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-16 00:56:23,839 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 42130 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-16 00:56:23,839 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HStore(1912): d538367277459b70f4a06235dfaacfec/C is initiating minor compaction (all files) 2023-04-16 00:56:23,840 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 111896 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-16 00:56:23,840 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HRegion(2259): Starting compaction of d538367277459b70f4a06235dfaacfec/C in TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. 2023-04-16 00:56:23,840 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HStore(1912): d538367277459b70f4a06235dfaacfec/A is initiating minor compaction (all files) 2023-04-16 00:56:23,840 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/8ab0b6d870784a6b9aa7ee9be8159c36, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/26bc504829a9431ba5e28f9fd59417f0, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/fa97bea0af4e4f5491453e35a38bd085, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/58df185bf1f14e7ebda08f4cc70d58d4] into tmpdir=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp, totalSize=41.1 K 2023-04-16 00:56:23,840 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of d538367277459b70f4a06235dfaacfec/A in TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. 2023-04-16 00:56:23,840 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A/fe26a1dd9f1a47328a6b6173cc96d91b, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A/da12a06b31d441d599d829b0ff669323, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A/22f90baaf5e44bbd86a35ea3cd725c23, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A/7582b7a5f2454c368c0ad8a4637a6015] into tmpdir=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp, totalSize=109.3 K 2023-04-16 00:56:23,840 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting 8ab0b6d870784a6b9aa7ee9be8159c36, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=347, earliestPutTs=1681606581223 2023-04-16 00:56:23,840 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.Compactor(207): Compacting fe26a1dd9f1a47328a6b6173cc96d91b, keycount=150, bloomtype=ROW, size=26.1 K, encoding=NONE, compression=NONE, seqNum=347, earliestPutTs=1681606581223 2023-04-16 00:56:23,841 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting 26bc504829a9431ba5e28f9fd59417f0, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=358, earliestPutTs=1681606581349 2023-04-16 00:56:23,841 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.Compactor(207): Compacting da12a06b31d441d599d829b0ff669323, keycount=100, bloomtype=ROW, size=18.6 K, encoding=NONE, compression=NONE, seqNum=358, earliestPutTs=1681606581349 2023-04-16 00:56:23,841 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting fa97bea0af4e4f5491453e35a38bd085, keycount=50, bloomtype=ROW, size=7.1 K, encoding=NONE, compression=NONE, seqNum=364, earliestPutTs=1681606582487 2023-04-16 00:56:23,841 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.Compactor(207): Compacting 22f90baaf5e44bbd86a35ea3cd725c23, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=364, earliestPutTs=1681606582468 2023-04-16 00:56:23,842 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting 58df185bf1f14e7ebda08f4cc70d58d4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=949, earliestPutTs=1681606583367 2023-04-16 00:56:23,842 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.Compactor(207): Compacting 7582b7a5f2454c368c0ad8a4637a6015, keycount=250, bloomtype=ROW, size=39.2 K, encoding=NONE, compression=NONE, seqNum=949, earliestPutTs=1681606583367 2023-04-16 00:56:23,849 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] throttle.PressureAwareThroughputController(145): d538367277459b70f4a06235dfaacfec#C#compaction#630 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-16 00:56:23,850 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] throttle.PressureAwareThroughputController(145): d538367277459b70f4a06235dfaacfec#A#compaction#629 average throughput is 6.57 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-16 00:56:23,856 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp/C/2a69af8ce41f4ef0a8a99865999bf042 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/2a69af8ce41f4ef0a8a99865999bf042 2023-04-16 00:56:23,861 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in d538367277459b70f4a06235dfaacfec/C of d538367277459b70f4a06235dfaacfec into 2a69af8ce41f4ef0a8a99865999bf042(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 00:56:23,861 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for d538367277459b70f4a06235dfaacfec: 2023-04-16 00:56:23,861 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec., storeName=d538367277459b70f4a06235dfaacfec/C, priority=12, startTime=1681606583838; duration=0sec 2023-04-16 00:56:23,861 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 00:56:23,861 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-16 00:56:23,862 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 42130 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-16 00:56:23,862 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HStore(1912): d538367277459b70f4a06235dfaacfec/B is initiating minor compaction (all files) 2023-04-16 00:56:23,863 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HRegion(2259): Starting compaction of d538367277459b70f4a06235dfaacfec/B in TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. 2023-04-16 00:56:23,863 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/000d29b1df9e4c9f8b4fd8726653f465, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/f62174af8c2047a3b8ad29743a771078, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/0454d7b6b16e44c79e571242d0b3a390, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/98f28f3e808b4fdabf1fa12b5c18e302] into tmpdir=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp, totalSize=41.1 K 2023-04-16 00:56:23,863 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting 000d29b1df9e4c9f8b4fd8726653f465, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=347, earliestPutTs=1681606581223 2023-04-16 00:56:23,863 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting f62174af8c2047a3b8ad29743a771078, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=358, earliestPutTs=1681606581349 2023-04-16 00:56:23,864 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting 0454d7b6b16e44c79e571242d0b3a390, keycount=50, bloomtype=ROW, size=7.1 K, encoding=NONE, compression=NONE, seqNum=364, earliestPutTs=1681606582487 2023-04-16 00:56:23,864 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting 98f28f3e808b4fdabf1fa12b5c18e302, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=949, earliestPutTs=1681606583367 2023-04-16 00:56:23,871 DEBUG [Thread-4138] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:56:23,872 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:56:23,873 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:56:23,881 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] throttle.PressureAwareThroughputController(145): d538367277459b70f4a06235dfaacfec#B#compaction#631 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-16 00:56:23,881 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:56:23,881 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:56:23,881 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:56:23,882 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:56:23,882 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:56:23,882 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:56:23,882 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:23,882 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:23,883 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:56:23,883 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:23,883 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:56:23,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:56:23,884 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:23,884 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:56:23,884 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp/A/19ac16e53a404a87a89e9f69b765373f as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A/19ac16e53a404a87a89e9f69b765373f 2023-04-16 00:56:23,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:23,885 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:23,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:56:23,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:23,885 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:56:23,888 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:56:23,888 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:56:23,888 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:56:23,888 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:56:23,888 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:56:23,888 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:56:23,888 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:56:23,888 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. 2023-04-16 00:56:23,889 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. started... 2023-04-16 00:56:23,889 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp/B/c5e34cd11dd64f82a0c80a478d39dc46 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/c5e34cd11dd64f82a0c80a478d39dc46 2023-04-16 00:56:23,889 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing d538367277459b70f4a06235dfaacfec 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-16 00:56:23,889 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d538367277459b70f4a06235dfaacfec, store=A 2023-04-16 00:56:23,889 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:56:23,889 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d538367277459b70f4a06235dfaacfec, store=B 2023-04-16 00:56:23,890 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:56:23,890 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d538367277459b70f4a06235dfaacfec, store=C 2023-04-16 00:56:23,890 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:56:23,890 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in d538367277459b70f4a06235dfaacfec/A of d538367277459b70f4a06235dfaacfec into 19ac16e53a404a87a89e9f69b765373f(size=26.2 K), total size for store is 26.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 00:56:23,890 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for d538367277459b70f4a06235dfaacfec: 2023-04-16 00:56:23,890 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec., storeName=d538367277459b70f4a06235dfaacfec/A, priority=12, startTime=1681606583838; duration=0sec 2023-04-16 00:56:23,891 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 00:56:23,894 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in d538367277459b70f4a06235dfaacfec/B of d538367277459b70f4a06235dfaacfec into c5e34cd11dd64f82a0c80a478d39dc46(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 00:56:23,894 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for d538367277459b70f4a06235dfaacfec: 2023-04-16 00:56:23,894 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec., storeName=d538367277459b70f4a06235dfaacfec/B, priority=12, startTime=1681606583838; duration=0sec 2023-04-16 00:56:23,894 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 00:56:23,905 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304168b83dcd486b74d6f94098f93777e1415 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304168b83dcd486b74d6f94098f93777e1415 2023-04-16 00:56:23,923 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(9158): Flush requested on d538367277459b70f4a06235dfaacfec 2023-04-16 00:56:23,923 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. as already flushing 2023-04-16 00:56:23,924 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=962, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp/A/f7aa501a91d443f9a493a7e3a88884b2 2023-04-16 00:56:23,932 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=962 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp/B/aa952594b8ee4cbd8d7653fcbb8da493 2023-04-16 00:56:23,972 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:23,972 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 304 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44548 deadline: 1681606643971, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:23,995 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:23,996 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 212 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44554 deadline: 1681606643994, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:23,998 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:23,999 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 208 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44564 deadline: 1681606643997, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:24,002 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:24,002 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 228 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44556 deadline: 1681606644001, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:24,002 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:24,002 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 249 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44562 deadline: 1681606644002, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:24,074 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:24,074 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 306 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44548 deadline: 1681606644073, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:24,276 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:24,276 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 308 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44548 deadline: 1681606644276, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:24,343 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=962 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp/C/6d70239dc59141e4b22c0a82755cf093 2023-04-16 00:56:24,346 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp/A/f7aa501a91d443f9a493a7e3a88884b2 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A/f7aa501a91d443f9a493a7e3a88884b2 2023-04-16 00:56:24,349 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A/f7aa501a91d443f9a493a7e3a88884b2, entries=100, sequenceid=962, filesize=18.6 K 2023-04-16 00:56:24,350 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp/B/aa952594b8ee4cbd8d7653fcbb8da493 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/aa952594b8ee4cbd8d7653fcbb8da493 2023-04-16 00:56:24,353 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/aa952594b8ee4cbd8d7653fcbb8da493, entries=100, sequenceid=962, filesize=9.5 K 2023-04-16 00:56:24,354 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp/C/6d70239dc59141e4b22c0a82755cf093 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/6d70239dc59141e4b22c0a82755cf093 2023-04-16 00:56:24,357 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/6d70239dc59141e4b22c0a82755cf093, entries=100, sequenceid=962, filesize=9.5 K 2023-04-16 00:56:24,357 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for d538367277459b70f4a06235dfaacfec in 468ms, sequenceid=962, compaction requested=false 2023-04-16 00:56:24,357 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for d538367277459b70f4a06235dfaacfec: 2023-04-16 00:56:24,358 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. 2023-04-16 00:56:24,358 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:56:24,358 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:56:24,358 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:24,358 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:56:24,358 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:56:24,360 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:24,360 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:24,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:24,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:24,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:24,360 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:24,361 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:56:24,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:24,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:24,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:24,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:24,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:24,362 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:56:24,362 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@148f892a[Count = 0] remaining members to acquire global barrier 2023-04-16 00:56:24,362 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:56:24,362 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:24,363 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:24,363 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:24,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:24,363 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:56:24,363 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:56:24,363 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:56:24,363 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:24,363 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:56:24,364 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:56:24,365 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:24,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:24,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:24,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:24,365 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:56:24,365 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:56:24,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:24,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:24,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:24,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:24,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:24,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:24,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:24,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:56:24,367 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:56:24,367 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:56:24,367 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:56:24,367 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:56:24,367 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:56:24,368 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:56:24,369 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:24,369 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:56:24,369 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:24,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:24,369 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:24,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:24,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:24,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:24,370 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:56:24,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:56:24,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:24,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:24,370 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:24,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:24,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:24,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:24,371 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:24,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:24,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:24,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:24,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:24,375 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:56:24,375 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:56:24,376 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:56:24,375 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:56:24,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:56:24,376 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:56:24,376 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:56:24,376 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:24,376 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:24,376 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:24,376 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:24,376 DEBUG [Thread-4138] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:56:24,376 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:56:24,376 DEBUG [Thread-4138] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:56:24,376 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:56:24,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:56:24,376 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:24,376 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:24,376 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:24,476 DEBUG [Thread-4138] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:56:24,477 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:56:24,478 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:56:24,480 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:56:24,480 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:56:24,480 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:56:24,480 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:56:24,480 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:56:24,480 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:56:24,480 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:24,480 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:24,482 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:24,482 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:56:24,482 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:56:24,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:56:24,482 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:24,482 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:56:24,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:24,483 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:24,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:56:24,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:24,484 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:56:24,484 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:56:24,485 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:56:24,485 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:56:24,485 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:56:24,485 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:56:24,485 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:56:24,485 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. 2023-04-16 00:56:24,485 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. started... 2023-04-16 00:56:24,485 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:56:24,485 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing d538367277459b70f4a06235dfaacfec 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-16 00:56:24,485 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d538367277459b70f4a06235dfaacfec, store=A 2023-04-16 00:56:24,485 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:56:24,485 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d538367277459b70f4a06235dfaacfec, store=B 2023-04-16 00:56:24,486 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:56:24,486 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d538367277459b70f4a06235dfaacfec, store=C 2023-04-16 00:56:24,486 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:56:24,498 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(9158): Flush requested on d538367277459b70f4a06235dfaacfec 2023-04-16 00:56:24,498 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. as already flushing 2023-04-16 00:56:24,502 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304161e7d0c225a344fa9945f392c73bf0ca2 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304161e7d0c225a344fa9945f392c73bf0ca2 2023-04-16 00:56:24,511 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:24,511 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 212 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44564 deadline: 1681606644509, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:24,511 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:24,512 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 232 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44556 deadline: 1681606644510, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:24,513 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:24,513 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 219 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44554 deadline: 1681606644511, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:24,513 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:24,514 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 252 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44562 deadline: 1681606644511, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:24,578 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:24,578 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 310 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44548 deadline: 1681606644577, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:24,613 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:24,613 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 214 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44564 deadline: 1681606644612, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:24,614 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:24,615 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 234 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44556 deadline: 1681606644612, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:24,616 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:24,617 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 221 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44554 deadline: 1681606644614, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:24,617 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:24,617 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 254 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44562 deadline: 1681606644614, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:24,816 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:24,817 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 216 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44564 deadline: 1681606644815, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:24,817 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:24,817 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 236 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44556 deadline: 1681606644817, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:24,819 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:24,819 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 223 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44554 deadline: 1681606644817, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:24,819 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:24,819 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 256 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44562 deadline: 1681606644819, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:24,914 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=984, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp/A/fc4225da4ed44d7a9492b896f4e9001b 2023-04-16 00:56:24,950 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=984 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp/B/10af9a31afdd44b2bb4c51e276db3426 2023-04-16 00:56:24,961 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=984 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp/C/4417e886af8f4637b7e2ecd8d778bf63 2023-04-16 00:56:24,965 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp/A/fc4225da4ed44d7a9492b896f4e9001b as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A/fc4225da4ed44d7a9492b896f4e9001b 2023-04-16 00:56:24,968 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A/fc4225da4ed44d7a9492b896f4e9001b, entries=150, sequenceid=984, filesize=25.4 K 2023-04-16 00:56:24,969 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp/B/10af9a31afdd44b2bb4c51e276db3426 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/10af9a31afdd44b2bb4c51e276db3426 2023-04-16 00:56:24,971 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/10af9a31afdd44b2bb4c51e276db3426, entries=150, sequenceid=984, filesize=11.9 K 2023-04-16 00:56:24,972 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp/C/4417e886af8f4637b7e2ecd8d778bf63 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/4417e886af8f4637b7e2ecd8d778bf63 2023-04-16 00:56:24,975 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/4417e886af8f4637b7e2ecd8d778bf63, entries=150, sequenceid=984, filesize=11.9 K 2023-04-16 00:56:24,975 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=67.09 KB/68700 for d538367277459b70f4a06235dfaacfec in 490ms, sequenceid=984, compaction requested=true 2023-04-16 00:56:24,975 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for d538367277459b70f4a06235dfaacfec: 2023-04-16 00:56:24,975 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. 2023-04-16 00:56:24,975 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:56:24,975 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:56:24,975 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:24,976 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:56:24,976 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:56:24,977 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:24,977 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:24,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:24,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:24,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:24,977 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:24,977 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:56:24,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:24,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:24,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:24,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:24,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:24,979 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:56:24,979 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@60227344[Count = 0] remaining members to acquire global barrier 2023-04-16 00:56:24,979 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:56:24,979 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:24,980 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:24,980 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:24,980 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:56:24,980 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:24,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:24,980 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:56:24,980 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:56:24,980 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:56:24,986 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:24,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:24,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:24,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:24,987 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:56:24,987 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:56:24,987 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:56:24,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:24,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:24,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:24,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:24,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:24,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:24,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:24,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:56:24,989 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:56:24,989 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:56:24,989 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:56:24,989 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:56:24,989 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:56:24,989 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:56:24,991 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:24,991 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:24,991 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:24,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:24,991 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:56:24,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:24,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:24,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:24,991 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:56:24,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:56:24,992 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:24,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:24,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:24,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:24,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:24,993 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:24,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:24,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:24,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:24,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:24,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:24,995 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:24,995 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:24,995 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:24,995 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:56:24,995 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:56:24,995 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:56:24,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:56:24,996 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:56:24,996 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:56:24,996 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:56:24,996 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:24,996 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:24,996 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:24,996 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:24,996 DEBUG [Thread-4138] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:56:24,996 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:56:24,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:56:24,996 DEBUG [Thread-4138] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:56:24,997 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:56:25,083 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(9158): Flush requested on d538367277459b70f4a06235dfaacfec 2023-04-16 00:56:25,083 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing d538367277459b70f4a06235dfaacfec 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-16 00:56:25,083 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d538367277459b70f4a06235dfaacfec, store=A 2023-04-16 00:56:25,083 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:56:25,083 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d538367277459b70f4a06235dfaacfec, store=B 2023-04-16 00:56:25,083 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:56:25,083 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d538367277459b70f4a06235dfaacfec, store=C 2023-04-16 00:56:25,083 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:56:25,096 DEBUG [Thread-4138] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:56:25,097 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:56:25,098 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:56:25,101 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:56:25,101 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:56:25,101 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:56:25,101 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:56:25,102 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:56:25,102 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:56:25,103 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:25,103 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:25,104 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:56:25,104 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:25,104 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:56:25,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:56:25,104 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:25,104 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:56:25,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:25,105 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:25,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:56:25,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:25,105 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:56:25,107 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:56:25,107 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:56:25,107 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:56:25,107 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:56:25,107 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:56:25,107 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:56:25,107 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:56:25,107 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. 2023-04-16 00:56:25,107 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. started... 2023-04-16 00:56:25,107 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. as already flushing 2023-04-16 00:56:25,107 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. 2023-04-16 00:56:25,107 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:56:25,108 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:56:25,108 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:25,108 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:56:25,108 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:56:25,110 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:25,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:25,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:25,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:25,110 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:25,111 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:25,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:25,111 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:56:25,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:25,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:25,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:25,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:25,112 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:56:25,112 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7a085a78[Count = 0] remaining members to acquire global barrier 2023-04-16 00:56:25,112 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:56:25,112 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:25,114 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:25,114 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:25,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:25,114 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:56:25,114 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:56:25,114 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:56:25,114 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:25,114 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:56:25,117 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230416057405dfbe9b46dfa9433776618779a1 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230416057405dfbe9b46dfa9433776618779a1 2023-04-16 00:56:25,121 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:56:25,121 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:56:25,121 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:56:25,128 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:25,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:25,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:25,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:25,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:25,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:25,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:25,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:25,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:25,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:25,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:25,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:56:25,131 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:56:25,131 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:56:25,131 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:56:25,131 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:56:25,131 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:56:25,131 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:56:25,133 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:25,133 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:25,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:25,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:25,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:25,133 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:25,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:25,133 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:56:25,133 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:25,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:25,133 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:56:25,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:56:25,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:25,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:25,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:25,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:25,134 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:25,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:25,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:25,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:25,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:25,136 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:25,136 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:25,136 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:56:25,136 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:56:25,136 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:25,136 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:56:25,136 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:56:25,136 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:25,136 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:56:25,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:56:25,136 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:56:25,136 DEBUG [Thread-4138] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:56:25,136 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:56:25,136 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:25,136 DEBUG [Thread-4138] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:56:25,137 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:56:25,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:56:25,137 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:25,137 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:25,148 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:25,148 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 225 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44554 deadline: 1681606645128, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:25,149 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:25,149 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 239 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44556 deadline: 1681606645146, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:25,149 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:25,149 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 219 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44564 deadline: 1681606645146, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:25,149 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:25,149 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 325 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44548 deadline: 1681606645146, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:25,150 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:25,150 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 259 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44562 deadline: 1681606645147, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:25,237 DEBUG [Thread-4138] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:56:25,237 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:56:25,239 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:56:25,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:56:25,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:56:25,240 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:56:25,240 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:56:25,240 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:56:25,240 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:56:25,241 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:25,241 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:25,243 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:56:25,243 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:25,243 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:56:25,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:56:25,243 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:25,243 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:56:25,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:25,244 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:25,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:56:25,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:25,244 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:56:25,244 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:56:25,244 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:56:25,244 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:56:25,244 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:56:25,244 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:56:25,245 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:56:25,245 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:56:25,245 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. 2023-04-16 00:56:25,245 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. started... 2023-04-16 00:56:25,245 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. as already flushing 2023-04-16 00:56:25,245 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. 2023-04-16 00:56:25,245 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:56:25,245 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:56:25,245 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:25,245 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:56:25,245 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:56:25,247 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:25,247 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:25,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:25,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:25,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:25,247 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:25,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:25,247 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:56:25,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:25,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:25,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:25,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:25,248 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:56:25,248 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5478cc8e[Count = 0] remaining members to acquire global barrier 2023-04-16 00:56:25,248 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:56:25,249 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:25,251 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:25,251 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 241 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44556 deadline: 1681606645250, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:25,251 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:25,251 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 327 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44548 deadline: 1681606645250, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:25,252 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:25,252 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 261 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44562 deadline: 1681606645251, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:25,254 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:25,254 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 221 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44564 deadline: 1681606645252, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:25,256 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:25,256 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:25,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:25,256 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:56:25,256 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:56:25,256 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:56:25,257 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:25,257 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:56:25,258 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:25,258 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:56:25,258 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:56:25,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:25,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:25,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:25,258 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:56:25,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:25,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:25,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:25,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:25,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:25,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:25,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:25,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:56:25,261 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:56:25,261 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:56:25,261 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:56:25,261 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:56:25,261 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:56:25,261 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:56:25,266 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:25,266 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:25,266 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:25,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:25,266 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:56:25,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:25,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:25,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:25,266 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:56:25,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:56:25,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:25,266 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:25,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:25,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:25,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:25,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:25,268 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:25,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:25,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:25,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:25,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:25,270 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:56:25,270 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:56:25,270 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:25,270 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:25,270 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:25,270 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:25,270 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:25,270 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:25,270 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:25,270 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:56:25,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:56:25,270 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:56:25,270 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:56:25,270 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:56:25,270 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:56:25,270 DEBUG [Thread-4138] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:56:25,270 DEBUG [Thread-4138] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:56:25,270 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:56:25,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:56:25,371 DEBUG [Thread-4138] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:56:25,371 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:56:25,373 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:56:25,374 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:56:25,374 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:56:25,374 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:56:25,374 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:56:25,374 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:56:25,374 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:56:25,374 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:25,375 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:25,376 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:56:25,376 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:25,376 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:56:25,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:56:25,376 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:25,376 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:56:25,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:25,377 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:25,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:56:25,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:25,377 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:56:25,377 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:56:25,377 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:56:25,377 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:56:25,377 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:56:25,377 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:56:25,378 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:56:25,378 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. 2023-04-16 00:56:25,378 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. started... 2023-04-16 00:56:25,378 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:56:25,378 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. as already flushing 2023-04-16 00:56:25,378 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. 2023-04-16 00:56:25,378 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:56:25,378 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:56:25,378 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:25,378 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:56:25,378 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:56:25,379 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:25,379 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:25,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:25,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:25,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:25,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:25,380 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:25,380 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:56:25,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:25,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:25,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:25,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:25,381 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:56:25,381 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3c56d9f2[Count = 0] remaining members to acquire global barrier 2023-04-16 00:56:25,381 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:56:25,381 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:25,383 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:25,383 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:25,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:25,383 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:56:25,383 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:56:25,383 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:56:25,383 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:25,383 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:56:25,384 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:25,384 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:56:25,384 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:56:25,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:25,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:25,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:25,384 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:56:25,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:25,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:25,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:25,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:25,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:25,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:25,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:25,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:56:25,387 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:56:25,387 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:56:25,387 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:56:25,387 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:56:25,387 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:56:25,387 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:56:25,388 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:25,388 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:25,388 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:56:25,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:25,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:25,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:25,388 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:25,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:25,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:25,388 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:25,389 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:56:25,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:56:25,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:25,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:25,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:25,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:25,390 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:25,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:25,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:25,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:25,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:25,391 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:25,391 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:25,391 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:56:25,391 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:56:25,392 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:56:25,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:56:25,392 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:56:25,391 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:25,392 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:56:25,392 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:25,392 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:25,392 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:25,392 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:25,392 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:56:25,392 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:56:25,392 DEBUG [Thread-4138] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:56:25,392 DEBUG [Thread-4138] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:56:25,392 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:56:25,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:56:25,453 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:25,454 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 243 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44556 deadline: 1681606645452, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:25,454 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:25,454 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 329 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44548 deadline: 1681606645452, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:25,454 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:25,454 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 263 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44562 deadline: 1681606645452, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:25,457 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:25,457 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 223 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44564 deadline: 1681606645455, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:25,492 DEBUG [Thread-4138] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:56:25,493 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:56:25,494 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:56:25,495 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:56:25,495 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:56:25,495 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:56:25,495 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:56:25,495 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:56:25,495 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:56:25,496 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:25,496 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:25,498 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:56:25,498 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:56:25,498 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:25,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:56:25,499 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:25,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:25,499 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:56:25,499 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:25,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:56:25,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:25,499 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:56:25,500 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:56:25,500 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:56:25,500 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:56:25,500 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:56:25,500 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:56:25,500 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:56:25,500 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:56:25,500 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. 2023-04-16 00:56:25,500 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. started... 2023-04-16 00:56:25,500 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. as already flushing 2023-04-16 00:56:25,500 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. 2023-04-16 00:56:25,500 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:56:25,500 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:56:25,500 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:25,500 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:56:25,500 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:56:25,502 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:25,502 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:25,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:25,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:25,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:25,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:25,502 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:25,502 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:56:25,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:25,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:25,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:25,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:25,503 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:56:25,503 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4f9d256f[Count = 0] remaining members to acquire global barrier 2023-04-16 00:56:25,503 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:56:25,503 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:25,505 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:25,505 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:25,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:25,505 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:56:25,505 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:56:25,505 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:25,505 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:56:25,505 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:56:25,506 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:56:25,506 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:25,506 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:56:25,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:25,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:25,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:25,506 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:56:25,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:25,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:25,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:25,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:25,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:25,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:25,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:25,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:56:25,509 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:56:25,509 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:56:25,509 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:56:25,509 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:56:25,509 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:56:25,509 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:56:25,511 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:25,511 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:25,511 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:56:25,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:25,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:25,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:25,511 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:25,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:25,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:25,512 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:56:25,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:56:25,512 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:25,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:25,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:25,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:25,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:25,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:25,513 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:25,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:25,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:25,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:25,519 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:25,519 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:56:25,519 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:25,519 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:56:25,519 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:56:25,519 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:56:25,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:56:25,519 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:56:25,519 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:56:25,519 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:25,519 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:56:25,519 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:25,519 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:25,519 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:25,519 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:56:25,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:56:25,519 DEBUG [Thread-4138] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:56:25,519 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:25,519 DEBUG [Thread-4138] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:56:25,530 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=998, memsize=24.6 K, hasBloomFilter=true, into tmp file hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp/A/d49c5ed4e0ff46a09e6ccf4ae0db5059 2023-04-16 00:56:25,538 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=998 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp/B/7ba5a16a30174fc09fe5b6e4ae542936 2023-04-16 00:56:25,546 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=998 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp/C/1617ddb85e8f4717942b9cc8524fba3f 2023-04-16 00:56:25,550 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp/A/d49c5ed4e0ff46a09e6ccf4ae0db5059 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A/d49c5ed4e0ff46a09e6ccf4ae0db5059 2023-04-16 00:56:25,553 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A/d49c5ed4e0ff46a09e6ccf4ae0db5059, entries=150, sequenceid=998, filesize=25.4 K 2023-04-16 00:56:25,554 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp/B/7ba5a16a30174fc09fe5b6e4ae542936 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/7ba5a16a30174fc09fe5b6e4ae542936 2023-04-16 00:56:25,556 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/7ba5a16a30174fc09fe5b6e4ae542936, entries=150, sequenceid=998, filesize=11.9 K 2023-04-16 00:56:25,557 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp/C/1617ddb85e8f4717942b9cc8524fba3f as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/1617ddb85e8f4717942b9cc8524fba3f 2023-04-16 00:56:25,560 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/1617ddb85e8f4717942b9cc8524fba3f, entries=150, sequenceid=998, filesize=11.9 K 2023-04-16 00:56:25,560 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for d538367277459b70f4a06235dfaacfec in 477ms, sequenceid=998, compaction requested=true 2023-04-16 00:56:25,560 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for d538367277459b70f4a06235dfaacfec: 2023-04-16 00:56:25,560 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 00:56:25,561 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 00:56:25,561 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-16 00:56:25,561 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-16 00:56:25,561 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 00:56:25,562 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 97954 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-16 00:56:25,562 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HStore(1912): d538367277459b70f4a06235dfaacfec/A is initiating minor compaction (all files) 2023-04-16 00:56:25,562 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of d538367277459b70f4a06235dfaacfec/A in TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. 2023-04-16 00:56:25,562 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47152 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-16 00:56:25,562 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HStore(1912): d538367277459b70f4a06235dfaacfec/B is initiating minor compaction (all files) 2023-04-16 00:56:25,562 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HRegion(2259): Starting compaction of d538367277459b70f4a06235dfaacfec/B in TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. 2023-04-16 00:56:25,562 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/c5e34cd11dd64f82a0c80a478d39dc46, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/aa952594b8ee4cbd8d7653fcbb8da493, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/10af9a31afdd44b2bb4c51e276db3426, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/7ba5a16a30174fc09fe5b6e4ae542936] into tmpdir=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp, totalSize=46.0 K 2023-04-16 00:56:25,562 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting c5e34cd11dd64f82a0c80a478d39dc46, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=949, earliestPutTs=1681606583367 2023-04-16 00:56:25,562 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A/19ac16e53a404a87a89e9f69b765373f, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A/f7aa501a91d443f9a493a7e3a88884b2, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A/fc4225da4ed44d7a9492b896f4e9001b, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A/d49c5ed4e0ff46a09e6ccf4ae0db5059] into tmpdir=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp, totalSize=95.7 K 2023-04-16 00:56:25,563 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting aa952594b8ee4cbd8d7653fcbb8da493, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=962, earliestPutTs=1681606583384 2023-04-16 00:56:25,563 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.Compactor(207): Compacting 19ac16e53a404a87a89e9f69b765373f, keycount=150, bloomtype=ROW, size=26.2 K, encoding=NONE, compression=NONE, seqNum=949, earliestPutTs=1681606583367 2023-04-16 00:56:25,563 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting 10af9a31afdd44b2bb4c51e276db3426, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=984, earliestPutTs=1681606583951 2023-04-16 00:56:25,563 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.Compactor(207): Compacting f7aa501a91d443f9a493a7e3a88884b2, keycount=100, bloomtype=ROW, size=18.6 K, encoding=NONE, compression=NONE, seqNum=962, earliestPutTs=1681606583384 2023-04-16 00:56:25,564 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting 7ba5a16a30174fc09fe5b6e4ae542936, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=998, earliestPutTs=1681606584504 2023-04-16 00:56:25,564 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.Compactor(207): Compacting fc4225da4ed44d7a9492b896f4e9001b, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=984, earliestPutTs=1681606583951 2023-04-16 00:56:25,564 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.Compactor(207): Compacting d49c5ed4e0ff46a09e6ccf4ae0db5059, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=998, earliestPutTs=1681606584504 2023-04-16 00:56:25,571 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] throttle.PressureAwareThroughputController(145): d538367277459b70f4a06235dfaacfec#B#compaction#641 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-16 00:56:25,578 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] throttle.PressureAwareThroughputController(145): d538367277459b70f4a06235dfaacfec#A#compaction#642 average throughput is 2.46 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-16 00:56:25,620 DEBUG [Thread-4138] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:56:25,620 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:56:25,621 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:56:25,623 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:56:25,623 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:56:25,623 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:56:25,623 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:56:25,623 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:56:25,623 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:56:25,623 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:25,623 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:25,625 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:56:25,625 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:25,625 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:56:25,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:56:25,625 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:25,625 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:56:25,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:25,625 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:25,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:56:25,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:25,626 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:56:25,626 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:56:25,626 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:56:25,626 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:56:25,626 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:56:25,626 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:56:25,626 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:56:25,626 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:56:25,626 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. 2023-04-16 00:56:25,626 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. started... 2023-04-16 00:56:25,626 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing d538367277459b70f4a06235dfaacfec 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-16 00:56:25,627 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d538367277459b70f4a06235dfaacfec, store=A 2023-04-16 00:56:25,627 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:56:25,627 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d538367277459b70f4a06235dfaacfec, store=B 2023-04-16 00:56:25,627 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:56:25,627 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d538367277459b70f4a06235dfaacfec, store=C 2023-04-16 00:56:25,627 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:56:25,654 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(9158): Flush requested on d538367277459b70f4a06235dfaacfec 2023-04-16 00:56:25,654 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. as already flushing 2023-04-16 00:56:25,693 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:25,693 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 238 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44554 deadline: 1681606645691, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:25,756 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:25,757 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 331 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44548 deadline: 1681606645755, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:25,757 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:25,757 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 265 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44562 deadline: 1681606645755, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:25,757 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:25,757 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 245 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44556 deadline: 1681606645756, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:25,761 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:25,761 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 225 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44564 deadline: 1681606645758, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:25,797 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:25,797 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 240 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44554 deadline: 1681606645794, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:25,983 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp/B/e917078e0ff64ebea0587c7db31a293a as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/e917078e0ff64ebea0587c7db31a293a 2023-04-16 00:56:25,986 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in d538367277459b70f4a06235dfaacfec/B of d538367277459b70f4a06235dfaacfec into e917078e0ff64ebea0587c7db31a293a(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 00:56:25,986 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for d538367277459b70f4a06235dfaacfec: 2023-04-16 00:56:25,986 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec., storeName=d538367277459b70f4a06235dfaacfec/B, priority=12, startTime=1681606585561; duration=0sec 2023-04-16 00:56:25,986 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 00:56:25,987 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-16 00:56:25,988 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47152 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-16 00:56:25,988 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HStore(1912): d538367277459b70f4a06235dfaacfec/C is initiating minor compaction (all files) 2023-04-16 00:56:25,988 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HRegion(2259): Starting compaction of d538367277459b70f4a06235dfaacfec/C in TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. 2023-04-16 00:56:25,988 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/2a69af8ce41f4ef0a8a99865999bf042, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/6d70239dc59141e4b22c0a82755cf093, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/4417e886af8f4637b7e2ecd8d778bf63, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/1617ddb85e8f4717942b9cc8524fba3f] into tmpdir=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp, totalSize=46.0 K 2023-04-16 00:56:25,988 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting 2a69af8ce41f4ef0a8a99865999bf042, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=949, earliestPutTs=1681606583367 2023-04-16 00:56:25,989 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting 6d70239dc59141e4b22c0a82755cf093, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=962, earliestPutTs=1681606583384 2023-04-16 00:56:25,989 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting 4417e886af8f4637b7e2ecd8d778bf63, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=984, earliestPutTs=1681606583951 2023-04-16 00:56:25,989 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting 1617ddb85e8f4717942b9cc8524fba3f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=998, earliestPutTs=1681606584504 2023-04-16 00:56:25,995 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] throttle.PressureAwareThroughputController(145): d538367277459b70f4a06235dfaacfec#C#compaction#644 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-16 00:56:26,001 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:26,001 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 242 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44554 deadline: 1681606645999, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:26,045 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023041697c671b16e084c7fb8d7c4a2b89d1b45 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041697c671b16e084c7fb8d7c4a2b89d1b45 2023-04-16 00:56:26,260 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:26,260 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 333 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44548 deadline: 1681606646259, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:26,261 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:26,261 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 267 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44562 deadline: 1681606646260, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:26,261 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:26,261 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 247 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44556 deadline: 1681606646260, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:26,264 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:26,264 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 227 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44564 deadline: 1681606646262, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:26,306 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:26,306 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 244 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44554 deadline: 1681606646304, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:26,404 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp/C/597fc8d38ece43eab4d4c09908966f14 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/597fc8d38ece43eab4d4c09908966f14 2023-04-16 00:56:26,410 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in d538367277459b70f4a06235dfaacfec/C of d538367277459b70f4a06235dfaacfec into 597fc8d38ece43eab4d4c09908966f14(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 00:56:26,410 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for d538367277459b70f4a06235dfaacfec: 2023-04-16 00:56:26,410 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec., storeName=d538367277459b70f4a06235dfaacfec/C, priority=12, startTime=1681606585561; duration=0sec 2023-04-16 00:56:26,410 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp/A/22db59fae0354e219d8e9c6da3fd024b as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A/22db59fae0354e219d8e9c6da3fd024b 2023-04-16 00:56:26,410 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 00:56:26,414 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in d538367277459b70f4a06235dfaacfec/A of d538367277459b70f4a06235dfaacfec into 22db59fae0354e219d8e9c6da3fd024b(size=26.4 K), total size for store is 26.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 00:56:26,414 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for d538367277459b70f4a06235dfaacfec: 2023-04-16 00:56:26,414 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec., storeName=d538367277459b70f4a06235dfaacfec/A, priority=12, startTime=1681606585560; duration=0sec 2023-04-16 00:56:26,414 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 00:56:26,449 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=1016, memsize=33.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp/A/8317419ec6ed413eb9152bb51d493d86 2023-04-16 00:56:26,459 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=1016 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp/B/c54e596b6b774e48b4ac80b98342a4fd 2023-04-16 00:56:26,468 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=1016 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp/C/f499850510f649b0963b8e24ed9becea 2023-04-16 00:56:26,472 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp/A/8317419ec6ed413eb9152bb51d493d86 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A/8317419ec6ed413eb9152bb51d493d86 2023-04-16 00:56:26,475 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A/8317419ec6ed413eb9152bb51d493d86, entries=150, sequenceid=1016, filesize=25.4 K 2023-04-16 00:56:26,476 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp/B/c54e596b6b774e48b4ac80b98342a4fd as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/c54e596b6b774e48b4ac80b98342a4fd 2023-04-16 00:56:26,479 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/c54e596b6b774e48b4ac80b98342a4fd, entries=150, sequenceid=1016, filesize=11.9 K 2023-04-16 00:56:26,479 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp/C/f499850510f649b0963b8e24ed9becea as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/f499850510f649b0963b8e24ed9becea 2023-04-16 00:56:26,482 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/f499850510f649b0963b8e24ed9becea, entries=150, sequenceid=1016, filesize=11.9 K 2023-04-16 00:56:26,483 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=73.80 KB/75570 for d538367277459b70f4a06235dfaacfec in 857ms, sequenceid=1016, compaction requested=false 2023-04-16 00:56:26,483 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for d538367277459b70f4a06235dfaacfec: 2023-04-16 00:56:26,483 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. 2023-04-16 00:56:26,483 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:56:26,483 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:56:26,483 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:26,483 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:56:26,483 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:56:26,484 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:26,485 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:26,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:26,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:26,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:26,485 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:26,485 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:56:26,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:26,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:26,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:26,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:26,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:26,487 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:56:26,487 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@77fe7d7e[Count = 0] remaining members to acquire global barrier 2023-04-16 00:56:26,487 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:56:26,487 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:26,488 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:26,488 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:26,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:26,489 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:56:26,489 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:56:26,489 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:56:26,489 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:26,489 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:56:26,490 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:26,490 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:56:26,490 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:56:26,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:26,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:26,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:26,490 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:56:26,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:26,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:26,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:26,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:26,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:26,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:26,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:26,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:56:26,492 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:56:26,492 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:56:26,492 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:56:26,492 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:56:26,492 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:56:26,492 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:56:26,495 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:26,495 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:26,495 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:56:26,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:26,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:26,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:26,495 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:26,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:26,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:26,496 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:56:26,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:56:26,496 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:26,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:26,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:26,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:26,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:26,497 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:26,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:26,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:26,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:26,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:26,500 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:26,500 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:26,500 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:56:26,500 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:56:26,500 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:56:26,500 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:26,500 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:56:26,500 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:56:26,500 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:56:26,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:56:26,500 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:56:26,500 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:26,500 DEBUG [Thread-4138] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:56:26,500 DEBUG [Thread-4138] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:56:26,500 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:26,500 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:26,500 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:26,501 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:56:26,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:56:26,600 DEBUG [Thread-4138] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:56:26,601 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:56:26,602 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:56:26,604 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:56:26,604 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:56:26,604 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:56:26,604 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:56:26,604 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:56:26,604 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:56:26,604 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:26,604 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:26,605 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:56:26,606 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:56:26,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:56:26,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:26,606 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:26,606 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:26,606 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:26,607 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:56:26,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:56:26,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:26,607 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:56:26,607 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:56:26,607 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:56:26,607 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:56:26,607 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:56:26,607 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:56:26,607 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:56:26,607 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:56:26,607 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. 2023-04-16 00:56:26,608 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. started... 2023-04-16 00:56:26,608 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing d538367277459b70f4a06235dfaacfec 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-16 00:56:26,608 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d538367277459b70f4a06235dfaacfec, store=A 2023-04-16 00:56:26,608 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:56:26,608 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d538367277459b70f4a06235dfaacfec, store=B 2023-04-16 00:56:26,608 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:56:26,608 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d538367277459b70f4a06235dfaacfec, store=C 2023-04-16 00:56:26,608 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:56:26,628 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230416ded766dc8fef4a4680a1e57235a514d3 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230416ded766dc8fef4a4680a1e57235a514d3 2023-04-16 00:56:26,808 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(9158): Flush requested on d538367277459b70f4a06235dfaacfec 2023-04-16 00:56:26,808 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. as already flushing 2023-04-16 00:56:26,849 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:26,849 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 261 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44554 deadline: 1681606646847, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:26,952 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:26,952 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 263 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44554 deadline: 1681606646950, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:27,034 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=1033, memsize=24.6 K, hasBloomFilter=true, into tmp file hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp/A/e242a6eb9be84c319cd7cd35d15f976f 2023-04-16 00:56:27,155 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:27,156 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 265 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44554 deadline: 1681606647153, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:27,266 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:27,266 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 269 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44562 deadline: 1681606647263, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:27,266 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:27,266 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 229 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44564 deadline: 1681606647266, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:27,269 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:27,270 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 335 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44548 deadline: 1681606647267, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:27,270 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:27,270 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 249 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44556 deadline: 1681606647267, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:27,446 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=1033 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp/B/c8f6e4654d98421c82dac933bc788866 2023-04-16 00:56:27,454 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=1033 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp/C/6707b49b39784d269b461831ac356ada 2023-04-16 00:56:27,457 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp/A/e242a6eb9be84c319cd7cd35d15f976f as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A/e242a6eb9be84c319cd7cd35d15f976f 2023-04-16 00:56:27,458 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:27,458 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 267 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44554 deadline: 1681606647457, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:27,461 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A/e242a6eb9be84c319cd7cd35d15f976f, entries=150, sequenceid=1033, filesize=25.4 K 2023-04-16 00:56:27,461 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp/B/c8f6e4654d98421c82dac933bc788866 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/c8f6e4654d98421c82dac933bc788866 2023-04-16 00:56:27,465 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/c8f6e4654d98421c82dac933bc788866, entries=150, sequenceid=1033, filesize=11.9 K 2023-04-16 00:56:27,466 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp/C/6707b49b39784d269b461831ac356ada as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/6707b49b39784d269b461831ac356ada 2023-04-16 00:56:27,469 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/6707b49b39784d269b461831ac356ada, entries=150, sequenceid=1033, filesize=11.9 K 2023-04-16 00:56:27,469 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for d538367277459b70f4a06235dfaacfec in 861ms, sequenceid=1033, compaction requested=true 2023-04-16 00:56:27,469 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for d538367277459b70f4a06235dfaacfec: 2023-04-16 00:56:27,470 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. 2023-04-16 00:56:27,470 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:56:27,470 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:56:27,470 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:27,470 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:56:27,470 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:56:27,471 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:27,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:27,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:27,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:27,472 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:27,472 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:27,472 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:56:27,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:27,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:27,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:27,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:27,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:27,473 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:56:27,473 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6570d3b3[Count = 0] remaining members to acquire global barrier 2023-04-16 00:56:27,473 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:56:27,473 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:27,476 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:27,476 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:27,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:27,476 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:56:27,476 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:56:27,476 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:56:27,476 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:27,476 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:56:27,478 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:27,478 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:56:27,478 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:56:27,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:27,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:27,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:27,478 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:56:27,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:27,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:27,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:27,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:27,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:27,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:27,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:27,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:56:27,480 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:56:27,480 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:56:27,480 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:56:27,480 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:56:27,480 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:56:27,480 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:56:27,482 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:27,482 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:27,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:27,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:27,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:27,482 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:27,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:27,482 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:56:27,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:27,482 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:56:27,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:56:27,483 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:27,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:27,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:27,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:27,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:27,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:27,484 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:27,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:27,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:27,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:27,486 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:56:27,486 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:27,486 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:56:27,486 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:56:27,486 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:56:27,486 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:56:27,486 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:27,486 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:27,486 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:27,486 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:27,486 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:56:27,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:56:27,486 DEBUG [Thread-4138] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:56:27,486 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:56:27,487 DEBUG [Thread-4138] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:56:27,486 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:27,487 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:27,487 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:56:27,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:56:27,587 DEBUG [Thread-4138] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:56:27,587 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:56:27,589 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:56:27,590 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:56:27,590 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:56:27,590 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:56:27,590 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:56:27,590 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:56:27,590 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:56:27,590 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:27,591 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:27,593 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:56:27,593 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:27,593 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:56:27,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:56:27,594 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:27,594 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:56:27,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:27,594 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:27,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:56:27,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:27,594 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:56:27,595 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:56:27,595 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:56:27,595 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:56:27,595 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:56:27,595 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:56:27,595 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:56:27,595 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:56:27,595 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. 2023-04-16 00:56:27,595 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. started... 2023-04-16 00:56:27,595 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing d538367277459b70f4a06235dfaacfec 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-16 00:56:27,595 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d538367277459b70f4a06235dfaacfec, store=A 2023-04-16 00:56:27,595 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:56:27,596 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d538367277459b70f4a06235dfaacfec, store=B 2023-04-16 00:56:27,596 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:56:27,596 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d538367277459b70f4a06235dfaacfec, store=C 2023-04-16 00:56:27,596 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:56:27,618 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023041670cc40e97c0846b8ba66c1aa1a26f06b to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041670cc40e97c0846b8ba66c1aa1a26f06b 2023-04-16 00:56:27,623 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=1051, memsize=33.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp/A/10cdf46636424ee38fafe671851b987e 2023-04-16 00:56:27,962 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(9158): Flush requested on d538367277459b70f4a06235dfaacfec 2023-04-16 00:56:27,962 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. as already flushing 2023-04-16 00:56:27,997 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:27,998 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 280 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44554 deadline: 1681606647995, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:28,031 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=1051 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp/B/c5d3816f8b4d4d149e50042d81e8ba7b 2023-04-16 00:56:28,041 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=1051 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp/C/405666d06d51430c89c17cabe8ad2ac3 2023-04-16 00:56:28,046 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp/A/10cdf46636424ee38fafe671851b987e as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A/10cdf46636424ee38fafe671851b987e 2023-04-16 00:56:28,049 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A/10cdf46636424ee38fafe671851b987e, entries=150, sequenceid=1051, filesize=25.4 K 2023-04-16 00:56:28,049 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp/B/c5d3816f8b4d4d149e50042d81e8ba7b as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/c5d3816f8b4d4d149e50042d81e8ba7b 2023-04-16 00:56:28,053 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/c5d3816f8b4d4d149e50042d81e8ba7b, entries=150, sequenceid=1051, filesize=11.9 K 2023-04-16 00:56:28,053 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp/C/405666d06d51430c89c17cabe8ad2ac3 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/405666d06d51430c89c17cabe8ad2ac3 2023-04-16 00:56:28,056 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/405666d06d51430c89c17cabe8ad2ac3, entries=150, sequenceid=1051, filesize=11.9 K 2023-04-16 00:56:28,057 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=73.80 KB/75570 for d538367277459b70f4a06235dfaacfec in 462ms, sequenceid=1051, compaction requested=true 2023-04-16 00:56:28,057 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for d538367277459b70f4a06235dfaacfec: 2023-04-16 00:56:28,057 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. 2023-04-16 00:56:28,057 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:56:28,057 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:56:28,057 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:28,058 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:56:28,058 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:56:28,059 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:28,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:28,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:28,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:28,060 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:28,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:28,060 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:28,060 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:56:28,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:28,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:28,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:28,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:28,061 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:56:28,061 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6fe31fa6[Count = 0] remaining members to acquire global barrier 2023-04-16 00:56:28,061 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:56:28,061 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:28,062 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:28,062 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:28,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:28,062 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:56:28,063 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:56:28,063 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:56:28,063 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:28,063 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:56:28,064 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:28,064 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:56:28,064 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:56:28,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:28,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:28,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:28,064 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:56:28,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:28,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:28,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:28,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:28,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:28,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:28,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:28,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:56:28,067 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:56:28,067 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:56:28,067 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:56:28,067 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:56:28,067 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:56:28,067 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:56:28,070 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:28,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:28,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:28,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:28,070 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:28,070 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:56:28,070 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:28,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:28,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:28,071 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:28,071 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:56:28,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:56:28,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:28,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:28,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:28,072 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:28,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:28,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:28,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:28,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:28,077 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:28,077 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:28,077 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:28,078 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:56:28,078 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:56:28,078 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:56:28,078 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:56:28,078 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:56:28,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:56:28,078 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:28,078 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:56:28,078 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:56:28,079 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:56:28,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:56:28,078 DEBUG [Thread-4138] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:56:28,079 DEBUG [Thread-4138] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:56:28,078 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:28,079 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:28,079 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:28,102 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(9158): Flush requested on d538367277459b70f4a06235dfaacfec 2023-04-16 00:56:28,102 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing d538367277459b70f4a06235dfaacfec 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-04-16 00:56:28,102 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d538367277459b70f4a06235dfaacfec, store=A 2023-04-16 00:56:28,102 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:56:28,102 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d538367277459b70f4a06235dfaacfec, store=B 2023-04-16 00:56:28,102 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:56:28,102 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d538367277459b70f4a06235dfaacfec, store=C 2023-04-16 00:56:28,102 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:56:28,114 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304161a68509eeba6424a86c07492a45b4c3a to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304161a68509eeba6424a86c07492a45b4c3a 2023-04-16 00:56:28,133 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=1066, memsize=26.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp/A/a11321a458434e23b0e455ab0c787fa5 2023-04-16 00:56:28,175 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:28,176 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 297 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44554 deadline: 1681606648173, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:28,179 DEBUG [Thread-4138] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:56:28,180 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:56:28,181 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:56:28,182 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:56:28,182 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:56:28,182 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:56:28,182 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:56:28,182 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:56:28,182 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:56:28,183 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:28,183 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:28,184 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:28,184 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:56:28,184 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:56:28,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:56:28,184 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:28,184 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:56:28,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:28,185 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:28,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:56:28,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:28,185 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:56:28,185 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:56:28,185 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:56:28,185 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:56:28,185 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:56:28,186 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:56:28,186 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:56:28,186 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. 2023-04-16 00:56:28,186 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. started... 2023-04-16 00:56:28,186 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. as already flushing 2023-04-16 00:56:28,186 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. 2023-04-16 00:56:28,186 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:56:28,186 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:56:28,186 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:28,186 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:56:28,186 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:56:28,187 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:28,187 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:28,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:28,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:28,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:28,188 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:28,188 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:56:28,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:28,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:28,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:28,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:28,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:28,189 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:56:28,189 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4973e968[Count = 0] remaining members to acquire global barrier 2023-04-16 00:56:28,189 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:56:28,189 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:28,193 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:28,193 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:28,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:28,193 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:56:28,193 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:56:28,193 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:56:28,194 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:28,194 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:56:28,195 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:28,195 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:56:28,195 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:56:28,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:28,195 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:56:28,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:28,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:28,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:28,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:28,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:28,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:28,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:28,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:28,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:28,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:56:28,198 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:56:28,198 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:56:28,198 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:56:28,198 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:56:28,198 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:56:28,198 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:56:28,199 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:28,199 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:56:28,199 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:28,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:28,199 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:28,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:28,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:28,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:28,200 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:56:28,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:56:28,200 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:28,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:28,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:28,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:28,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:28,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:28,201 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:28,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:28,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:28,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:28,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:28,203 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:28,203 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:56:28,204 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:28,204 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:56:28,204 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:28,204 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:56:28,204 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:56:28,204 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:56:28,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:56:28,204 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:56:28,204 DEBUG [Thread-4138] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:56:28,204 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:28,204 DEBUG [Thread-4138] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:56:28,204 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:56:28,204 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:28,204 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:56:28,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:56:28,204 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:28,204 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:28,277 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:28,278 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 299 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44554 deadline: 1681606648277, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:28,304 DEBUG [Thread-4138] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:56:28,305 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:56:28,306 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:56:28,307 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:56:28,308 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:56:28,308 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:56:28,308 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:56:28,308 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:56:28,308 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:56:28,308 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:28,308 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:28,310 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:56:28,310 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:28,310 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:56:28,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:56:28,311 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:28,311 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:56:28,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:28,311 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:28,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:56:28,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:28,311 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:56:28,312 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:56:28,312 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:56:28,312 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:56:28,312 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:56:28,312 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:56:28,312 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:56:28,312 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:56:28,312 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. 2023-04-16 00:56:28,312 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. started... 2023-04-16 00:56:28,312 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. as already flushing 2023-04-16 00:56:28,312 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. 2023-04-16 00:56:28,312 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:56:28,312 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:56:28,312 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:28,312 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:56:28,312 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:56:28,314 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:28,314 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:28,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:28,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:28,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:28,314 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:28,314 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:56:28,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:28,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:28,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:28,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:28,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:28,316 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:56:28,316 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4de268b5[Count = 0] remaining members to acquire global barrier 2023-04-16 00:56:28,316 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:56:28,316 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:28,317 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:28,317 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:28,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:28,317 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:56:28,317 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:56:28,317 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:56:28,317 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:28,317 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:56:28,318 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:28,318 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:56:28,318 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:56:28,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:28,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:28,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:28,318 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:56:28,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:28,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:28,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:28,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:28,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:28,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:28,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:28,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:56:28,320 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:56:28,320 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:56:28,320 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:56:28,320 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:56:28,320 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:56:28,321 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:56:28,323 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:28,323 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:28,323 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:28,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:28,323 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:56:28,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:28,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:28,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:28,323 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:56:28,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:56:28,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:28,323 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:28,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:28,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:28,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:28,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:28,325 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:28,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:28,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:28,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:28,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:28,327 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:56:28,327 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:56:28,327 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:28,327 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:56:28,327 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:56:28,327 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:56:28,327 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:56:28,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:56:28,327 DEBUG [Thread-4138] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:56:28,327 DEBUG [Thread-4138] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:56:28,327 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:56:28,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:56:28,327 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:28,327 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:56:28,328 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:28,328 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:28,328 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:28,328 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:28,328 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:28,427 DEBUG [Thread-4138] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:56:28,428 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:56:28,430 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:56:28,431 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:56:28,431 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:56:28,431 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:56:28,431 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:56:28,431 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:56:28,431 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:56:28,431 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:28,431 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:28,433 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:56:28,433 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:28,433 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:56:28,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:56:28,433 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:28,433 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:56:28,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:28,433 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:28,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:56:28,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:28,434 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:56:28,434 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:56:28,434 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:56:28,434 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:56:28,434 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:56:28,434 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:56:28,434 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:56:28,434 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:56:28,434 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. 2023-04-16 00:56:28,434 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. started... 2023-04-16 00:56:28,435 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. as already flushing 2023-04-16 00:56:28,435 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. 2023-04-16 00:56:28,435 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:56:28,435 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:56:28,435 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:28,435 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:56:28,435 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:56:28,436 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:28,436 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:28,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:28,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:28,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:28,436 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:28,436 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:56:28,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:28,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:28,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:28,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:28,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:28,438 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:56:28,438 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2e6cc7f1[Count = 0] remaining members to acquire global barrier 2023-04-16 00:56:28,438 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:56:28,438 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:28,440 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:28,440 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:28,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:28,440 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:56:28,440 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:56:28,440 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:56:28,440 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:28,440 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:56:28,441 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:56:28,441 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:56:28,441 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:28,441 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:56:28,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:28,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:28,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:28,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:28,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:28,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:28,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:28,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:28,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:28,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:28,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:56:28,444 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:56:28,444 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:56:28,444 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:56:28,444 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:56:28,444 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:56:28,444 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:56:28,445 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:28,445 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:28,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:28,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:28,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:28,445 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:28,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:28,445 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:56:28,446 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:56:28,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:56:28,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:28,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:28,446 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:28,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:28,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:28,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:28,447 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:28,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:28,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:28,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:28,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:28,449 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:56:28,449 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:28,449 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:56:28,450 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:56:28,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:56:28,450 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:56:28,450 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:56:28,449 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:28,450 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:28,450 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:28,450 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:28,450 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:28,450 DEBUG [Thread-4138] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:56:28,450 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:28,449 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:56:28,450 DEBUG [Thread-4138] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:56:28,450 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:56:28,450 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:56:28,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:56:28,481 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:28,481 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 301 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44554 deadline: 1681606648479, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:28,550 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=1066 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp/B/36121c3b7ff04cf09f73986d1ff097e3 2023-04-16 00:56:28,550 DEBUG [Thread-4138] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:56:28,551 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:56:28,553 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:56:28,554 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:56:28,554 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:56:28,554 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:56:28,554 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:56:28,554 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:56:28,555 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:56:28,555 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:28,555 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:28,560 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:56:28,560 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:28,560 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:56:28,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:56:28,560 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:28,560 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:56:28,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:28,561 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:28,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:56:28,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:28,561 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:56:28,561 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:56:28,561 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:56:28,561 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:56:28,561 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:56:28,561 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:56:28,562 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:56:28,562 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:56:28,562 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. 2023-04-16 00:56:28,562 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. started... 2023-04-16 00:56:28,562 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. as already flushing 2023-04-16 00:56:28,562 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. 2023-04-16 00:56:28,562 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:56:28,562 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:56:28,562 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:28,562 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:56:28,562 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:56:28,564 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:28,564 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:28,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:28,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:28,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:28,564 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:28,564 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:56:28,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:28,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:28,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:28,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:28,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:28,566 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:56:28,566 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5985f3b5[Count = 0] remaining members to acquire global barrier 2023-04-16 00:56:28,566 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:56:28,566 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:28,567 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:28,567 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:28,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:28,567 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:56:28,567 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:56:28,567 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:28,567 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:56:28,567 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:56:28,569 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:28,570 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:56:28,570 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:56:28,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:28,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:28,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:28,570 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:56:28,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:28,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:28,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:28,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:28,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:28,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:28,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:28,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:56:28,573 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:56:28,573 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:56:28,573 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:56:28,573 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:56:28,573 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:56:28,573 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:56:28,575 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:28,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:28,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:28,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:28,575 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:28,575 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:56:28,575 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:28,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:28,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:28,575 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:28,575 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:56:28,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:56:28,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:28,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:28,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:28,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:28,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:28,577 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:28,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:28,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:28,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:28,579 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:28,579 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:28,579 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:28,579 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:28,579 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:28,579 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:56:28,579 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:56:28,579 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:56:28,579 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:56:28,579 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:28,579 DEBUG [Thread-4138] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:56:28,579 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:56:28,579 DEBUG [Thread-4138] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:56:28,579 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:28,579 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:56:28,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:56:28,579 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:56:28,580 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:56:28,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:56:28,680 DEBUG [Thread-4138] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:56:28,680 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:56:28,681 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:56:28,682 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:56:28,682 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:56:28,682 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:56:28,682 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:56:28,682 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:56:28,682 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:56:28,683 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:28,683 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:28,684 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:56:28,684 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:28,684 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:56:28,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:56:28,684 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:28,684 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:56:28,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:28,685 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:28,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:56:28,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:28,685 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:56:28,685 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:56:28,685 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:56:28,685 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:56:28,685 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:56:28,685 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:56:28,685 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:56:28,685 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:56:28,685 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. 2023-04-16 00:56:28,685 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. started... 2023-04-16 00:56:28,686 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. as already flushing 2023-04-16 00:56:28,686 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. 2023-04-16 00:56:28,686 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:56:28,686 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:56:28,686 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:28,686 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:56:28,686 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:56:28,688 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:28,688 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:28,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:28,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:28,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:28,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:28,689 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:28,689 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:56:28,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:28,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:28,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:28,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:28,690 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:56:28,690 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@feb7944[Count = 0] remaining members to acquire global barrier 2023-04-16 00:56:28,690 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:56:28,690 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:28,691 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:28,691 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:28,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:28,691 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:56:28,691 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:56:28,691 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:28,691 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:56:28,691 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:56:28,692 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:56:28,692 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:56:28,692 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:28,692 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:56:28,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:28,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:28,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:28,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:28,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:28,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:28,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:28,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:28,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:28,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:28,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:56:28,694 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:56:28,694 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:56:28,694 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:56:28,694 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:56:28,694 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:56:28,694 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:56:28,696 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:28,696 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:28,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:28,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:28,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:28,696 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:28,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:28,696 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:56:28,696 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:56:28,696 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:28,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:56:28,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:28,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:28,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:28,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:28,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:28,698 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:28,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:28,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:28,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:28,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:28,700 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:28,700 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:28,700 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:28,700 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:28,700 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:56:28,700 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:56:28,701 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:56:28,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:56:28,700 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:28,701 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:56:28,701 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:56:28,700 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:56:28,701 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:28,701 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:28,701 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:56:28,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:56:28,701 DEBUG [Thread-4138] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:56:28,701 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:56:28,701 DEBUG [Thread-4138] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:56:28,784 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:28,784 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 303 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44554 deadline: 1681606648782, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:28,801 DEBUG [Thread-4138] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:56:28,802 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:56:28,803 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:56:28,804 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:56:28,804 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:56:28,804 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:56:28,805 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:56:28,805 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:56:28,805 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:56:28,805 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:28,805 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:28,806 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:56:28,806 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:28,806 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:56:28,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:56:28,807 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:28,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:28,807 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:56:28,807 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:28,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:56:28,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:28,808 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:56:28,808 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:56:28,808 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:56:28,808 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:56:28,808 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:56:28,808 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:56:28,808 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:56:28,808 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. 2023-04-16 00:56:28,808 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. started... 2023-04-16 00:56:28,808 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:56:28,808 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. as already flushing 2023-04-16 00:56:28,809 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. 2023-04-16 00:56:28,809 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:56:28,809 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:56:28,809 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:28,809 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:56:28,809 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:56:28,810 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:28,810 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:28,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:28,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:28,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:28,810 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:28,810 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:56:28,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:28,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:28,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:28,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:28,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:28,813 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:56:28,813 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@51120fcc[Count = 0] remaining members to acquire global barrier 2023-04-16 00:56:28,813 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:56:28,813 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:28,814 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:28,814 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:28,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:28,814 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:56:28,814 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:56:28,814 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:28,814 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:56:28,814 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:56:28,817 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:56:28,817 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:56:28,817 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:28,817 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:56:28,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:28,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:28,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:28,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:28,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:28,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:28,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:28,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:28,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:28,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:28,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:56:28,819 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:56:28,819 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:56:28,819 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:56:28,820 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:56:28,820 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:56:28,820 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:56:28,821 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:28,821 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:28,821 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:56:28,821 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:28,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:28,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:28,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:28,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:28,821 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:56:28,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:56:28,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:28,822 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:28,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:28,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:28,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:28,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:28,823 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:28,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:28,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:28,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:28,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:28,825 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:56:28,826 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:56:28,826 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:56:28,826 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:28,826 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:28,826 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:56:28,826 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:56:28,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:56:28,826 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:28,826 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:56:28,826 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:56:28,826 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:28,826 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:56:28,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:56:28,826 DEBUG [Thread-4138] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:56:28,826 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:28,826 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:28,826 DEBUG [Thread-4138] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:56:28,826 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:28,926 DEBUG [Thread-4138] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:56:28,927 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:56:28,929 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:56:28,930 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:56:28,930 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:56:28,930 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:56:28,930 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:56:28,930 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:56:28,930 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:56:28,930 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:28,930 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:28,932 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:56:28,932 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:28,932 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:56:28,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:56:28,932 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:28,932 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:56:28,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:28,933 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:28,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:56:28,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:28,933 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:56:28,933 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:56:28,933 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:56:28,933 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:56:28,933 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:56:28,933 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:56:28,934 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:56:28,934 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:56:28,934 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. 2023-04-16 00:56:28,934 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. started... 2023-04-16 00:56:28,934 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. as already flushing 2023-04-16 00:56:28,934 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. 2023-04-16 00:56:28,934 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:56:28,934 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:56:28,934 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:28,934 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:56:28,934 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:56:28,939 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:28,939 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:28,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:28,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:28,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:28,939 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:28,939 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:56:28,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:28,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:28,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:28,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:28,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:28,942 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:56:28,942 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@19090950[Count = 0] remaining members to acquire global barrier 2023-04-16 00:56:28,942 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:56:28,942 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:28,943 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:28,943 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:28,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:28,943 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:28,943 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:56:28,943 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:56:28,943 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:56:28,943 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:56:28,945 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:56:28,945 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:56:28,945 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:28,945 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:56:28,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:28,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:28,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:28,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:28,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:28,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:28,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:28,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:28,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:28,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:28,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:56:28,947 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:56:28,947 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:56:28,947 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:56:28,947 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:56:28,947 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:56:28,947 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:56:28,949 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:28,949 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:28,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:28,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:28,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:28,949 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:28,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:28,949 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:56:28,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:28,950 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:28,950 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:56:28,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:56:28,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:28,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:28,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:28,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:28,951 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:28,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:28,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:28,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:28,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:28,955 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:28,955 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:28,955 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:28,955 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:56:28,955 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:56:28,955 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:56:28,955 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:56:28,955 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:28,955 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:56:28,955 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:28,955 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:56:28,955 DEBUG [Thread-4138] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:56:28,955 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:56:28,955 DEBUG [Thread-4138] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:56:28,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:56:28,955 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:28,956 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:28,956 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:56:28,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:56:28,963 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=1066 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp/C/75e75e6448fb4b3faebf20cf9b1e65e3 2023-04-16 00:56:28,966 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp/A/a11321a458434e23b0e455ab0c787fa5 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A/a11321a458434e23b0e455ab0c787fa5 2023-04-16 00:56:28,970 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A/a11321a458434e23b0e455ab0c787fa5, entries=200, sequenceid=1066, filesize=32.3 K 2023-04-16 00:56:28,971 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp/B/36121c3b7ff04cf09f73986d1ff097e3 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/36121c3b7ff04cf09f73986d1ff097e3 2023-04-16 00:56:28,975 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/36121c3b7ff04cf09f73986d1ff097e3, entries=150, sequenceid=1066, filesize=11.9 K 2023-04-16 00:56:28,975 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp/C/75e75e6448fb4b3faebf20cf9b1e65e3 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/75e75e6448fb4b3faebf20cf9b1e65e3 2023-04-16 00:56:28,978 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/75e75e6448fb4b3faebf20cf9b1e65e3, entries=150, sequenceid=1066, filesize=11.9 K 2023-04-16 00:56:28,979 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=93.93 KB/96180 for d538367277459b70f4a06235dfaacfec in 877ms, sequenceid=1066, compaction requested=true 2023-04-16 00:56:28,979 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for d538367277459b70f4a06235dfaacfec: 2023-04-16 00:56:28,979 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 00:56:28,979 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-16 00:56:28,979 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-16 00:56:28,979 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 00:56:28,979 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 00:56:28,981 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61918 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-16 00:56:28,981 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HStore(1912): d538367277459b70f4a06235dfaacfec/B is initiating minor compaction (all files) 2023-04-16 00:56:28,981 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HRegion(2259): Starting compaction of d538367277459b70f4a06235dfaacfec/B in TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. 2023-04-16 00:56:28,981 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/e917078e0ff64ebea0587c7db31a293a, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/c54e596b6b774e48b4ac80b98342a4fd, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/c8f6e4654d98421c82dac933bc788866, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/c5d3816f8b4d4d149e50042d81e8ba7b, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/36121c3b7ff04cf09f73986d1ff097e3] into tmpdir=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp, totalSize=60.5 K 2023-04-16 00:56:28,981 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 138224 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-16 00:56:28,981 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HStore(1912): d538367277459b70f4a06235dfaacfec/A is initiating minor compaction (all files) 2023-04-16 00:56:28,982 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of d538367277459b70f4a06235dfaacfec/A in TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. 2023-04-16 00:56:28,982 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting e917078e0ff64ebea0587c7db31a293a, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=998, earliestPutTs=1681606584504 2023-04-16 00:56:28,982 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A/22db59fae0354e219d8e9c6da3fd024b, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A/8317419ec6ed413eb9152bb51d493d86, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A/e242a6eb9be84c319cd7cd35d15f976f, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A/10cdf46636424ee38fafe671851b987e, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A/a11321a458434e23b0e455ab0c787fa5] into tmpdir=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp, totalSize=135.0 K 2023-04-16 00:56:28,982 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting c54e596b6b774e48b4ac80b98342a4fd, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1016, earliestPutTs=1681606585117 2023-04-16 00:56:28,982 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.Compactor(207): Compacting 22db59fae0354e219d8e9c6da3fd024b, keycount=150, bloomtype=ROW, size=26.4 K, encoding=NONE, compression=NONE, seqNum=998, earliestPutTs=1681606584504 2023-04-16 00:56:28,982 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting c8f6e4654d98421c82dac933bc788866, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1033, earliestPutTs=1681606585669 2023-04-16 00:56:28,982 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.Compactor(207): Compacting 8317419ec6ed413eb9152bb51d493d86, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=1016, earliestPutTs=1681606585117 2023-04-16 00:56:28,982 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting c5d3816f8b4d4d149e50042d81e8ba7b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1051, earliestPutTs=1681606586829 2023-04-16 00:56:28,983 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.Compactor(207): Compacting e242a6eb9be84c319cd7cd35d15f976f, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=1033, earliestPutTs=1681606585669 2023-04-16 00:56:28,983 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting 36121c3b7ff04cf09f73986d1ff097e3, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1066, earliestPutTs=1681606587980 2023-04-16 00:56:28,983 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.Compactor(207): Compacting 10cdf46636424ee38fafe671851b987e, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=1051, earliestPutTs=1681606586829 2023-04-16 00:56:28,983 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.Compactor(207): Compacting a11321a458434e23b0e455ab0c787fa5, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=1066, earliestPutTs=1681606587980 2023-04-16 00:56:29,000 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] throttle.PressureAwareThroughputController(145): d538367277459b70f4a06235dfaacfec#B#compaction#656 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-16 00:56:29,009 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] throttle.PressureAwareThroughputController(145): d538367277459b70f4a06235dfaacfec#A#compaction#657 average throughput is 1.79 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-16 00:56:29,026 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp/B/e21e02618d174623b613a8d1163f365f as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/e21e02618d174623b613a8d1163f365f 2023-04-16 00:56:29,031 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in d538367277459b70f4a06235dfaacfec/B of d538367277459b70f4a06235dfaacfec into e21e02618d174623b613a8d1163f365f(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 00:56:29,031 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for d538367277459b70f4a06235dfaacfec: 2023-04-16 00:56:29,031 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec., storeName=d538367277459b70f4a06235dfaacfec/B, priority=11, startTime=1681606588979; duration=0sec 2023-04-16 00:56:29,031 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 00:56:29,032 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-16 00:56:29,035 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61918 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-16 00:56:29,035 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HStore(1912): d538367277459b70f4a06235dfaacfec/C is initiating minor compaction (all files) 2023-04-16 00:56:29,035 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HRegion(2259): Starting compaction of d538367277459b70f4a06235dfaacfec/C in TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. 2023-04-16 00:56:29,035 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/597fc8d38ece43eab4d4c09908966f14, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/f499850510f649b0963b8e24ed9becea, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/6707b49b39784d269b461831ac356ada, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/405666d06d51430c89c17cabe8ad2ac3, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/75e75e6448fb4b3faebf20cf9b1e65e3] into tmpdir=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp, totalSize=60.5 K 2023-04-16 00:56:29,036 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting 597fc8d38ece43eab4d4c09908966f14, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=998, earliestPutTs=1681606584504 2023-04-16 00:56:29,036 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting f499850510f649b0963b8e24ed9becea, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1016, earliestPutTs=1681606585117 2023-04-16 00:56:29,037 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting 6707b49b39784d269b461831ac356ada, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1033, earliestPutTs=1681606585669 2023-04-16 00:56:29,037 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting 405666d06d51430c89c17cabe8ad2ac3, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1051, earliestPutTs=1681606586829 2023-04-16 00:56:29,037 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting 75e75e6448fb4b3faebf20cf9b1e65e3, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1066, earliestPutTs=1681606587980 2023-04-16 00:56:29,039 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp/A/1122a71abf6a45af95f9c3a6185f56fe as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A/1122a71abf6a45af95f9c3a6185f56fe 2023-04-16 00:56:29,043 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in d538367277459b70f4a06235dfaacfec/A of d538367277459b70f4a06235dfaacfec into 1122a71abf6a45af95f9c3a6185f56fe(size=26.5 K), total size for store is 26.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 00:56:29,044 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for d538367277459b70f4a06235dfaacfec: 2023-04-16 00:56:29,044 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec., storeName=d538367277459b70f4a06235dfaacfec/A, priority=11, startTime=1681606588979; duration=0sec 2023-04-16 00:56:29,044 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 00:56:29,046 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] throttle.PressureAwareThroughputController(145): d538367277459b70f4a06235dfaacfec#C#compaction#658 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-16 00:56:29,055 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp/C/1ca76d69bd8841bd947f25033ae474d3 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/1ca76d69bd8841bd947f25033ae474d3 2023-04-16 00:56:29,056 DEBUG [Thread-4138] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:56:29,056 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:56:29,058 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:56:29,058 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in d538367277459b70f4a06235dfaacfec/C of d538367277459b70f4a06235dfaacfec into 1ca76d69bd8841bd947f25033ae474d3(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 00:56:29,058 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for d538367277459b70f4a06235dfaacfec: 2023-04-16 00:56:29,058 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec., storeName=d538367277459b70f4a06235dfaacfec/C, priority=11, startTime=1681606588979; duration=0sec 2023-04-16 00:56:29,059 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 00:56:29,059 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:56:29,059 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:56:29,059 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:56:29,059 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:56:29,059 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:56:29,059 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:56:29,060 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:29,060 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:29,061 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:56:29,061 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:29,061 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:56:29,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:56:29,062 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:29,062 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:56:29,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:29,062 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:29,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:56:29,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:29,063 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:56:29,063 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:56:29,063 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:56:29,063 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:56:29,063 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:56:29,063 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:56:29,063 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:56:29,063 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. 2023-04-16 00:56:29,063 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:56:29,063 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. started... 2023-04-16 00:56:29,063 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing d538367277459b70f4a06235dfaacfec 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-04-16 00:56:29,064 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d538367277459b70f4a06235dfaacfec, store=A 2023-04-16 00:56:29,064 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:56:29,064 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d538367277459b70f4a06235dfaacfec, store=B 2023-04-16 00:56:29,064 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:56:29,064 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d538367277459b70f4a06235dfaacfec, store=C 2023-04-16 00:56:29,064 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:56:29,084 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304166cd68c43b0cc4fd8abd2648397431db4 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304166cd68c43b0cc4fd8abd2648397431db4 2023-04-16 00:56:29,088 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=1086, memsize=31.3 K, hasBloomFilter=true, into tmp file hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp/A/a5afc425ea0b4a36acfd05a67e740914 2023-04-16 00:56:29,097 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=1086 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp/B/1b14d0512fb54b2ca3808faeebf965a6 2023-04-16 00:56:29,108 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=1086 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp/C/e4ed3fb509e94ee294acb2f9a36aada0 2023-04-16 00:56:29,112 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp/A/a5afc425ea0b4a36acfd05a67e740914 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A/a5afc425ea0b4a36acfd05a67e740914 2023-04-16 00:56:29,115 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A/a5afc425ea0b4a36acfd05a67e740914, entries=150, sequenceid=1086, filesize=25.4 K 2023-04-16 00:56:29,116 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp/B/1b14d0512fb54b2ca3808faeebf965a6 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/1b14d0512fb54b2ca3808faeebf965a6 2023-04-16 00:56:29,119 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/1b14d0512fb54b2ca3808faeebf965a6, entries=150, sequenceid=1086, filesize=11.9 K 2023-04-16 00:56:29,119 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp/C/e4ed3fb509e94ee294acb2f9a36aada0 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/e4ed3fb509e94ee294acb2f9a36aada0 2023-04-16 00:56:29,123 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/e4ed3fb509e94ee294acb2f9a36aada0, entries=150, sequenceid=1086, filesize=11.9 K 2023-04-16 00:56:29,125 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=0 B/0 for d538367277459b70f4a06235dfaacfec in 62ms, sequenceid=1086, compaction requested=false 2023-04-16 00:56:29,125 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for d538367277459b70f4a06235dfaacfec: 2023-04-16 00:56:29,126 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. 2023-04-16 00:56:29,126 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:56:29,126 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:56:29,126 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:29,126 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:56:29,126 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:56:29,127 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:29,128 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:29,128 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:56:29,129 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:29,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:29,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:29,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:29,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:29,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:29,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:29,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:29,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:29,131 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:56:29,131 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6fe7599f[Count = 0] remaining members to acquire global barrier 2023-04-16 00:56:29,131 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:56:29,131 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:29,132 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:29,132 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:29,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:29,132 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:56:29,132 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:56:29,132 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:56:29,132 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:29,132 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:56:29,134 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:29,134 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:56:29,135 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:56:29,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:29,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:29,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:29,135 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:56:29,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:29,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:29,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:29,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:29,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:29,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:29,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:29,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:56:29,137 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:56:29,137 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:56:29,137 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:56:29,137 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:56:29,137 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:56:29,137 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:56:29,139 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:29,139 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:56:29,139 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:29,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:29,139 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:29,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:29,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:29,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:29,139 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:56:29,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:56:29,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:29,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:29,140 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:29,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:29,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:29,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:29,141 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:29,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:29,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:29,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:29,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:29,143 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:56:29,143 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:29,143 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:56:29,143 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:56:29,143 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:56:29,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:56:29,143 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:56:29,143 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:29,143 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:56:29,143 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:29,143 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:29,143 DEBUG [Thread-4138] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:56:29,144 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:56:29,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:56:29,143 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:56:29,144 DEBUG [Thread-4138] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:56:29,143 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:29,144 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:29,144 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:29,244 DEBUG [Thread-4138] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:56:29,244 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:56:29,246 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:56:29,247 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:56:29,247 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:56:29,247 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:56:29,247 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:56:29,247 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:56:29,247 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:56:29,248 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:29,248 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:29,249 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:29,249 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:56:29,249 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:56:29,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:56:29,249 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:29,249 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:56:29,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:29,250 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:29,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:56:29,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:29,250 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:56:29,250 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:56:29,251 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:56:29,251 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:56:29,251 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:56:29,251 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:56:29,251 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:56:29,251 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:56:29,251 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. 2023-04-16 00:56:29,251 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. started... 2023-04-16 00:56:29,251 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for d538367277459b70f4a06235dfaacfec: 2023-04-16 00:56:29,251 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. 2023-04-16 00:56:29,251 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:56:29,251 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:56:29,251 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:29,251 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:56:29,252 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:56:29,254 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:29,254 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:29,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:29,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:29,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:29,254 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:29,254 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:56:29,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:29,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:29,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:29,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:29,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:29,256 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:56:29,256 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3b122c7e[Count = 0] remaining members to acquire global barrier 2023-04-16 00:56:29,256 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:56:29,256 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:29,257 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:29,257 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:29,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:29,257 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:56:29,257 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:56:29,257 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:56:29,257 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:29,257 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:56:29,258 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:56:29,258 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:56:29,258 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:29,258 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:56:29,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:29,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:29,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:29,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:29,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:29,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:29,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:29,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:29,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:29,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:29,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:56:29,261 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:56:29,261 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:56:29,261 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:56:29,261 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:56:29,261 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:56:29,261 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:56:29,262 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:29,262 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:56:29,262 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:29,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:29,262 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:29,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:29,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:29,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:29,263 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:29,263 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:56:29,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:56:29,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:29,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:29,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:29,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:29,264 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:29,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:29,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:29,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:29,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:29,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:29,267 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:29,267 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:56:29,267 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:56:29,267 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:56:29,267 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:56:29,267 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:56:29,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:56:29,268 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:56:29,268 DEBUG [Thread-4138] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:56:29,267 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:56:29,267 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:29,268 DEBUG [Thread-4138] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:56:29,268 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:56:29,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:56:29,268 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:29,268 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:29,268 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:29,268 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:29,268 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:29,300 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(9158): Flush requested on d538367277459b70f4a06235dfaacfec 2023-04-16 00:56:29,300 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing d538367277459b70f4a06235dfaacfec 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-16 00:56:29,301 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d538367277459b70f4a06235dfaacfec, store=A 2023-04-16 00:56:29,302 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:56:29,302 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d538367277459b70f4a06235dfaacfec, store=B 2023-04-16 00:56:29,302 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:56:29,302 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d538367277459b70f4a06235dfaacfec, store=C 2023-04-16 00:56:29,302 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:56:29,316 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230416e7c7683bacf849e68c5a0cbe7d77f264 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230416e7c7683bacf849e68c5a0cbe7d77f264 2023-04-16 00:56:29,321 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:29,321 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 276 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44562 deadline: 1681606649318, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:29,322 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:29,322 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 342 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44548 deadline: 1681606649319, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:29,322 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681606589311, max timestamp=1681606589320Num uniques 859; 2023-04-16 00:56:29,322 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681606589311, max timestamp=1681606589320Num uniques 859; 2023-04-16 00:56:29,322 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681606589311, max timestamp=1681606589320Num uniques 859; 2023-04-16 00:56:29,323 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-16 00:56:29,322 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-16 00:56:29,323 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-16 00:56:29,366 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:29,367 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 315 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44554 deadline: 1681606649362, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:29,367 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:29,367 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:29,367 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 242 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44564 deadline: 1681606649363, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:29,367 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 261 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44556 deadline: 1681606649362, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:29,368 DEBUG [Thread-4138] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:56:29,368 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:56:29,370 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:56:29,371 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:56:29,371 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:56:29,371 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:56:29,371 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:56:29,371 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:56:29,371 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:56:29,371 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:29,371 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:29,373 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:56:29,373 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:56:29,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:56:29,373 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:29,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:29,373 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:29,374 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:29,374 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:56:29,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:56:29,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:29,374 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:56:29,374 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:56:29,374 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:56:29,374 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:56:29,374 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:56:29,374 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:56:29,375 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:56:29,375 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:56:29,375 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. 2023-04-16 00:56:29,375 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. started... 2023-04-16 00:56:29,375 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. as already flushing 2023-04-16 00:56:29,375 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. 2023-04-16 00:56:29,375 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:56:29,375 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:56:29,375 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:29,375 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:56:29,375 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:56:29,385 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:29,385 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:29,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:29,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:29,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:29,386 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:29,386 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:56:29,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:29,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:29,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:29,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:29,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:29,387 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:56:29,387 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7a3f38bf[Count = 0] remaining members to acquire global barrier 2023-04-16 00:56:29,387 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:56:29,387 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:29,388 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:29,388 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:29,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:29,388 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:56:29,388 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:56:29,389 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:56:29,389 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:29,389 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:56:29,396 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:56:29,396 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:56:29,396 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:29,396 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:56:29,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:29,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:29,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:29,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:29,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:29,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:29,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:29,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:29,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:29,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:29,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:56:29,399 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:56:29,399 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:56:29,399 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:56:29,399 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:56:29,399 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:56:29,399 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:56:29,401 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:29,401 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:29,401 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:29,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:29,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:29,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:29,401 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:56:29,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:29,402 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:56:29,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:56:29,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:29,402 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:29,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:29,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:29,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:29,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:29,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:29,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:29,404 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:29,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:29,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:29,406 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:56:29,406 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:56:29,406 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:56:29,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:56:29,406 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:29,406 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:56:29,406 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:29,406 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:56:29,406 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:56:29,406 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:29,406 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:29,406 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:29,406 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:29,406 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:29,407 DEBUG [Thread-4138] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:56:29,407 DEBUG [Thread-4138] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:56:29,407 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:56:29,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:56:29,424 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:29,424 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 278 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44562 deadline: 1681606649422, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:29,425 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:29,425 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 344 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44548 deadline: 1681606649423, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:29,469 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:29,469 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 317 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44554 deadline: 1681606649468, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:29,470 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:29,470 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 244 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44564 deadline: 1681606649468, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:29,470 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:29,470 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 263 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44556 deadline: 1681606649468, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:29,507 DEBUG [Thread-4138] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:56:29,507 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:56:29,510 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:56:29,512 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:56:29,512 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:56:29,512 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:56:29,512 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:56:29,512 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:56:29,512 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:56:29,512 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:29,513 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:29,514 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:56:29,514 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:29,514 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:56:29,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:56:29,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:29,514 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:29,515 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:29,515 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:56:29,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:56:29,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:29,515 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:56:29,515 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:56:29,515 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:56:29,515 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:56:29,515 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:56:29,515 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:56:29,516 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:56:29,516 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:56:29,516 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. 2023-04-16 00:56:29,516 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. started... 2023-04-16 00:56:29,516 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. as already flushing 2023-04-16 00:56:29,516 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. 2023-04-16 00:56:29,516 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:56:29,516 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:56:29,516 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:29,517 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:56:29,517 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:56:29,520 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:29,520 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:29,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:29,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:29,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:29,521 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:29,521 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:56:29,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:29,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:29,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:29,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:29,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:29,523 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:56:29,523 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2baded21[Count = 0] remaining members to acquire global barrier 2023-04-16 00:56:29,523 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:56:29,523 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:29,524 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:29,524 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:29,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:29,524 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:56:29,524 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:56:29,524 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:56:29,524 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:29,524 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:56:29,525 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:56:29,525 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:56:29,525 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:29,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:29,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:29,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:29,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:29,525 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:56:29,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:29,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:29,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:29,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:29,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:29,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:29,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:56:29,529 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:56:29,529 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:56:29,529 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:56:29,529 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:56:29,529 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:56:29,529 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:56:29,532 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:29,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:29,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:29,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:29,532 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:29,532 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:56:29,532 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:29,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:29,532 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:56:29,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:56:29,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:29,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:29,533 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:29,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:29,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:29,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:29,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:29,540 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:29,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:29,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:29,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:29,544 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:56:29,544 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:56:29,544 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:56:29,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:56:29,544 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:29,544 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:29,544 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:29,544 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:29,544 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:29,544 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:56:29,544 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:29,545 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:56:29,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:56:29,544 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:56:29,545 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:56:29,545 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:29,545 DEBUG [Thread-4138] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:56:29,545 DEBUG [Thread-4138] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:56:29,629 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:29,629 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 346 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44548 deadline: 1681606649627, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:29,629 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:29,629 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 280 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44562 deadline: 1681606649627, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:29,646 DEBUG [Thread-4138] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:56:29,646 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:56:29,648 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:56:29,649 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:56:29,649 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:56:29,649 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:56:29,649 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:56:29,649 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:56:29,649 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:56:29,650 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:29,650 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:29,651 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:56:29,651 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:29,651 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:56:29,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:56:29,652 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:29,652 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:56:29,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:29,652 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:29,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:56:29,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:29,653 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:56:29,653 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:56:29,653 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:56:29,653 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:56:29,653 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:56:29,653 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:56:29,660 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:56:29,660 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. 2023-04-16 00:56:29,660 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. started... 2023-04-16 00:56:29,660 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:56:29,660 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. as already flushing 2023-04-16 00:56:29,660 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. 2023-04-16 00:56:29,660 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:56:29,660 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:56:29,660 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:29,660 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:56:29,660 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:56:29,662 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:29,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:29,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:29,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:29,662 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:29,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:29,662 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:29,662 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:56:29,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:29,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:29,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:29,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:29,664 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:56:29,664 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7ed3626a[Count = 0] remaining members to acquire global barrier 2023-04-16 00:56:29,664 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:56:29,664 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:29,665 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:29,666 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:29,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:29,666 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:56:29,666 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:56:29,666 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:56:29,666 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:29,666 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:56:29,668 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:29,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:29,669 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:56:29,669 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:56:29,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:29,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:29,669 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:56:29,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:29,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:29,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:29,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:29,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:29,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:29,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:29,672 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:29,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:56:29,672 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 319 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44554 deadline: 1681606649671, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:29,672 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:56:29,672 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:56:29,672 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:56:29,672 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:56:29,672 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:56:29,672 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:56:29,674 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:29,674 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:29,674 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:29,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:29,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:29,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:29,674 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:29,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:29,674 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:56:29,674 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 246 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44564 deadline: 1681606649671, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:29,675 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:29,675 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:56:29,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:56:29,675 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 265 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44556 deadline: 1681606649672, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:29,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:29,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:29,676 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:29,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:29,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:29,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:29,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:29,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:29,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:29,678 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:29,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:29,680 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:29,681 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:29,681 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:29,681 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:56:29,681 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:29,681 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:56:29,681 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:56:29,681 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:29,681 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:29,681 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:56:29,681 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:29,681 DEBUG [Thread-4138] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:56:29,681 DEBUG [Thread-4138] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:56:29,681 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:56:29,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:56:29,681 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:56:29,682 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:56:29,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:56:29,721 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=1098, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp/A/d97367c08cde484ba1badf53f5a0366f 2023-04-16 00:56:29,781 DEBUG [Thread-4138] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:56:29,782 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:56:29,784 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:56:29,785 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:56:29,785 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:56:29,785 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:56:29,785 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:56:29,786 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:56:29,786 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:56:29,786 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:29,786 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:29,787 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:29,787 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:56:29,787 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:56:29,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:56:29,788 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:29,788 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:56:29,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:29,788 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:29,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:56:29,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:29,788 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:56:29,789 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:56:29,789 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:56:29,789 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:56:29,789 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:56:29,789 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:56:29,792 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:56:29,792 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:56:29,792 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. 2023-04-16 00:56:29,792 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. started... 2023-04-16 00:56:29,792 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. as already flushing 2023-04-16 00:56:29,792 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. 2023-04-16 00:56:29,792 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:56:29,792 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:56:29,793 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:29,793 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:56:29,793 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:56:29,795 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:29,795 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:29,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:29,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:29,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:29,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:29,795 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:29,795 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:56:29,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:29,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:29,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:29,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:29,797 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:56:29,797 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5488c012[Count = 0] remaining members to acquire global barrier 2023-04-16 00:56:29,797 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:56:29,797 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:29,798 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:29,798 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:29,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:29,798 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:56:29,798 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:56:29,798 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:56:29,798 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:29,798 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:56:29,800 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:56:29,800 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:56:29,800 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:56:29,800 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:29,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:29,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:29,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:29,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:29,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:29,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:29,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:29,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:29,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:29,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:29,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:56:29,803 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:56:29,803 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:56:29,803 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:56:29,803 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:56:29,803 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:56:29,803 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:56:29,808 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:29,808 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:29,808 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:56:29,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:29,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:29,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:29,808 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:29,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:29,808 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:56:29,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:29,809 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:29,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:56:29,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:29,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:29,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:29,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:29,810 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:29,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:29,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:29,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:29,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:29,812 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:29,812 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:56:29,812 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:56:29,812 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:56:29,812 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:56:29,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:56:29,812 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:56:29,812 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:56:29,812 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:56:29,812 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:29,812 DEBUG [Thread-4138] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:56:29,813 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:29,813 DEBUG [Thread-4138] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:56:29,813 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:29,813 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:29,813 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:29,813 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:29,813 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:56:29,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:56:29,913 DEBUG [Thread-4138] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:56:29,913 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:56:29,915 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:56:29,916 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:56:29,916 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:56:29,916 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:56:29,916 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:56:29,916 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:56:29,916 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:56:29,917 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:29,917 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:29,918 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:56:29,918 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:29,918 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:56:29,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:56:29,918 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:29,918 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:56:29,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:29,919 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:29,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:56:29,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:29,919 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:56:29,920 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:56:29,920 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:56:29,920 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:56:29,920 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:56:29,920 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:56:29,920 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:56:29,920 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. 2023-04-16 00:56:29,920 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. started... 2023-04-16 00:56:29,920 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:56:29,920 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. as already flushing 2023-04-16 00:56:29,920 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. 2023-04-16 00:56:29,921 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:56:29,921 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:56:29,921 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:29,921 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:56:29,921 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:56:29,922 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:29,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:29,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:29,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:29,922 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:29,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:29,923 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:29,923 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:56:29,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:29,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:29,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:29,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:29,924 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:56:29,924 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5771f0fc[Count = 0] remaining members to acquire global barrier 2023-04-16 00:56:29,924 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:56:29,925 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:29,926 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:29,926 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:29,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:29,927 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:56:29,927 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:29,927 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:56:29,927 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:56:29,927 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:56:29,928 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:29,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:29,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:29,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:29,928 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:56:29,929 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:56:29,929 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:56:29,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:29,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:29,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:29,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:29,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:29,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:29,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:29,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:56:29,931 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:56:29,931 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:56:29,931 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:56:29,932 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:56:29,932 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:56:29,932 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:56:29,933 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:29,933 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:29,934 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:56:29,934 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:29,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:29,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:29,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:29,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:29,934 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:29,934 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:29,934 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 348 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44548 deadline: 1681606649932, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:29,934 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:56:29,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:56:29,934 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:29,934 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 282 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44562 deadline: 1681606649932, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:29,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:29,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:29,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:29,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:29,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:29,936 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:29,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:29,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:29,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:29,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:29,938 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:29,938 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:29,938 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:29,938 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:29,939 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:29,939 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:29,938 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:56:29,939 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:29,939 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:56:29,939 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:56:29,939 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:56:29,939 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:56:29,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:56:29,939 DEBUG [Thread-4138] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:56:29,939 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:56:29,939 DEBUG [Thread-4138] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:56:29,939 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:56:29,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:56:29,939 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:56:29,977 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:29,977 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 321 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44554 deadline: 1681606649975, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:29,978 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:29,978 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 267 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44556 deadline: 1681606649977, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:29,979 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:29,979 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 248 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44564 deadline: 1681606649977, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:30,039 DEBUG [Thread-4138] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:56:30,040 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:56:30,041 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:56:30,042 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:56:30,042 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:56:30,043 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:56:30,043 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:56:30,043 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:56:30,043 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:56:30,043 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:30,043 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:30,045 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:56:30,045 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:30,045 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:56:30,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:56:30,046 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:30,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:30,046 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:56:30,046 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:30,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:56:30,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:30,046 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:56:30,046 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:56:30,046 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:56:30,047 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:56:30,047 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:56:30,047 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:56:30,047 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:56:30,047 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. 2023-04-16 00:56:30,047 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. started... 2023-04-16 00:56:30,047 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:56:30,047 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. as already flushing 2023-04-16 00:56:30,047 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. 2023-04-16 00:56:30,047 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:56:30,047 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:56:30,047 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:30,047 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:56:30,047 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:56:30,049 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:30,049 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:30,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:30,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:30,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:30,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:30,049 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:30,049 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:56:30,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:30,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:30,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:30,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:30,050 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:56:30,050 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@13f1dbd6[Count = 0] remaining members to acquire global barrier 2023-04-16 00:56:30,050 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:56:30,050 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:30,052 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:30,052 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:30,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:30,052 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:56:30,052 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:56:30,052 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:30,052 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:56:30,052 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:56:30,053 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:56:30,053 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:56:30,053 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:30,053 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:56:30,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:30,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:30,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:30,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:30,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:30,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:30,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:30,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:30,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:30,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:30,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:56:30,056 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:56:30,056 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:56:30,056 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:56:30,056 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:56:30,056 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:56:30,056 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:56:30,059 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:30,060 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:30,063 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:56:30,063 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:56:30,063 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:56:30,063 DEBUG [Thread-4138] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:56:30,063 DEBUG [Thread-4138] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:56:30,064 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:30,064 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:56:30,064 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:56:30,064 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:30,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:30,064 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:30,064 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:30,064 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:30,064 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:30,064 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:30,064 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:30,064 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:30,064 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:30,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:30,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:30,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:30,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:30,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:30,065 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:56:30,065 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:56:30,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:56:30,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:30,065 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:56:30,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:56:30,066 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:56:30,132 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=1098 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp/B/c95993956ef44794ab016c69f0c94946 2023-04-16 00:56:30,163 DEBUG [Thread-4138] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:56:30,164 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:56:30,165 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:56:30,166 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:56:30,166 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:56:30,166 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:56:30,166 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:56:30,167 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:56:30,167 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:56:30,167 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:30,167 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:30,168 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:56:30,168 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:56:30,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:56:30,168 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:30,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:30,169 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:30,169 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:56:30,169 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:30,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:56:30,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:30,170 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:56:30,170 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:56:30,170 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:56:30,170 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:56:30,170 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:56:30,170 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:56:30,170 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. 2023-04-16 00:56:30,171 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. started... 2023-04-16 00:56:30,170 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:56:30,171 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. as already flushing 2023-04-16 00:56:30,171 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. 2023-04-16 00:56:30,171 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:56:30,171 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:56:30,171 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:30,171 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:56:30,171 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:56:30,173 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:30,173 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:30,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:30,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:30,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:30,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:30,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:30,174 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:30,174 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:56:30,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:30,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:30,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:30,175 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:56:30,175 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@76d42bf0[Count = 0] remaining members to acquire global barrier 2023-04-16 00:56:30,175 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:56:30,175 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:30,177 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:30,177 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:30,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:30,177 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:56:30,177 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:56:30,177 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:56:30,177 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:30,177 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:56:30,179 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:30,179 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:56:30,179 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:56:30,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:30,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:30,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:30,179 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:56:30,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:30,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:30,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:30,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:30,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:30,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:30,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:30,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:56:30,181 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:56:30,182 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:56:30,182 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:56:30,182 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:56:30,182 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:56:30,182 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:56:30,183 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:30,183 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:56:30,183 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:30,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:30,183 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:30,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:30,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:30,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:30,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:30,184 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:56:30,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:56:30,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:30,184 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:30,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:30,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:30,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:30,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:30,186 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:30,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:30,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:30,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:30,197 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:30,197 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:56:30,197 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:30,197 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:56:30,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:56:30,197 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:56:30,198 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:56:30,197 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:30,197 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:56:30,198 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:30,198 DEBUG [Thread-4138] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:56:30,198 DEBUG [Thread-4138] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:56:30,198 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:56:30,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:56:30,197 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:56:30,198 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:30,198 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:30,198 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:30,298 DEBUG [Thread-4138] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:56:30,298 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:56:30,300 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:56:30,301 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:56:30,301 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:56:30,301 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:56:30,301 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:56:30,301 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:56:30,301 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:56:30,301 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:30,302 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:30,303 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:56:30,303 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:30,303 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:56:30,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:56:30,304 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:30,304 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:56:30,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:30,304 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:30,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:56:30,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:30,304 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:56:30,305 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:56:30,305 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:56:30,305 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:56:30,305 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:56:30,305 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:56:30,305 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:56:30,305 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. 2023-04-16 00:56:30,305 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. started... 2023-04-16 00:56:30,305 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:56:30,305 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. as already flushing 2023-04-16 00:56:30,305 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. 2023-04-16 00:56:30,305 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:56:30,305 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:56:30,305 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:30,305 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:56:30,305 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:56:30,306 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:30,306 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:30,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:30,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:30,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:30,307 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:30,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:30,307 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:56:30,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:30,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:30,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:30,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:30,308 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:56:30,308 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@644eed68[Count = 0] remaining members to acquire global barrier 2023-04-16 00:56:30,308 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:56:30,309 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:30,311 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:30,311 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:30,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:30,311 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:56:30,311 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:56:30,311 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:56:30,311 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:30,311 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:56:30,313 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:30,313 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:56:30,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:30,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:30,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:30,313 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:56:30,313 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:56:30,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:30,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:30,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:30,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:30,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:30,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:30,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:30,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:56:30,316 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:56:30,316 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:56:30,316 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:56:30,316 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:56:30,316 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:56:30,316 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:56:30,317 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:30,317 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:30,317 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:30,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:30,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:30,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:30,317 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:56:30,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:30,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:30,318 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:56:30,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:56:30,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:30,318 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:30,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:30,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:30,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:30,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:30,319 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:30,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:30,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:30,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:30,322 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:56:30,322 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:30,322 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:56:30,322 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:56:30,322 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:56:30,322 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:56:30,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:56:30,322 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:56:30,322 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:56:30,322 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:30,323 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:30,323 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:30,323 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:30,323 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:30,323 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:30,323 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:56:30,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:56:30,323 DEBUG [Thread-4138] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:56:30,323 DEBUG [Thread-4138] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:56:30,424 DEBUG [Thread-4138] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:56:30,424 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:56:30,426 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:56:30,427 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:56:30,427 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:56:30,427 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:56:30,427 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:56:30,427 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:56:30,427 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:56:30,427 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:30,427 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:30,428 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:56:30,428 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:56:30,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:56:30,428 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:30,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:30,429 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:30,429 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:56:30,429 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:30,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:56:30,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:30,429 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:56:30,430 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:56:30,430 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:56:30,430 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:56:30,430 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:56:30,430 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:56:30,430 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:56:30,430 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:56:30,430 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. 2023-04-16 00:56:30,430 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. started... 2023-04-16 00:56:30,430 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. as already flushing 2023-04-16 00:56:30,430 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. 2023-04-16 00:56:30,430 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:56:30,430 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:56:30,431 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:30,431 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:56:30,431 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:56:30,433 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:30,433 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:30,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:30,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:30,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:30,433 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:30,433 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:56:30,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:30,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:30,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:30,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:30,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:30,434 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:56:30,435 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3c60170d[Count = 0] remaining members to acquire global barrier 2023-04-16 00:56:30,435 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:56:30,435 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:30,436 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:30,436 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:30,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:30,436 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:56:30,436 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:56:30,436 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:56:30,436 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:30,436 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:56:30,437 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:56:30,437 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:56:30,437 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:30,437 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:56:30,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:30,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:30,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:30,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:30,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:30,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:30,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:30,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:30,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:30,439 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:30,439 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 284 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44562 deadline: 1681606650436, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:30,439 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:30,439 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 350 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44548 deadline: 1681606650437, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:30,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:30,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:56:30,440 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:56:30,440 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:56:30,440 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:56:30,440 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:56:30,440 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:56:30,440 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:56:30,442 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:30,442 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:30,442 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:30,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:30,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:30,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:30,442 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:56:30,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:30,442 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:56:30,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:56:30,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:30,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:30,443 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:30,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:30,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:30,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:30,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:30,444 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:30,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:30,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:30,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:30,447 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:56:30,447 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:56:30,447 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:56:30,447 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:30,447 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:56:30,447 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:56:30,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:56:30,447 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:56:30,447 DEBUG [Thread-4138] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:56:30,447 DEBUG [Thread-4138] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:56:30,447 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:30,447 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:30,448 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:30,448 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:30,448 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:30,448 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:30,447 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:56:30,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:56:30,480 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:30,481 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 323 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44554 deadline: 1681606650478, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:30,485 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:30,485 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:30,485 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 250 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44564 deadline: 1681606650482, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:30,485 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 269 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44556 deadline: 1681606650483, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:30,542 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=1098 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp/C/27642fd671a84c598fc9bba20080fbb3 2023-04-16 00:56:30,546 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp/A/d97367c08cde484ba1badf53f5a0366f as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A/d97367c08cde484ba1badf53f5a0366f 2023-04-16 00:56:30,548 DEBUG [Thread-4138] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:56:30,548 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:56:30,550 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:56:30,550 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A/d97367c08cde484ba1badf53f5a0366f, entries=350, sequenceid=1098, filesize=52.9 K 2023-04-16 00:56:30,551 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:56:30,551 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:56:30,551 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:56:30,551 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:56:30,551 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:56:30,551 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:56:30,551 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp/B/c95993956ef44794ab016c69f0c94946 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/c95993956ef44794ab016c69f0c94946 2023-04-16 00:56:30,552 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:30,552 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:30,553 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:56:30,553 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:30,553 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:56:30,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:56:30,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:30,553 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:30,553 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:56:30,554 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:30,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:56:30,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:30,554 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:56:30,554 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:56:30,554 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:56:30,554 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:56:30,554 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:56:30,554 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:56:30,555 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:56:30,555 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:56:30,555 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. 2023-04-16 00:56:30,555 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/c95993956ef44794ab016c69f0c94946, entries=150, sequenceid=1098, filesize=11.9 K 2023-04-16 00:56:30,555 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. started... 2023-04-16 00:56:30,555 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. as already flushing 2023-04-16 00:56:30,555 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. 2023-04-16 00:56:30,555 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:56:30,555 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:56:30,555 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:30,556 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:56:30,556 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:56:30,556 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp/C/27642fd671a84c598fc9bba20080fbb3 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/27642fd671a84c598fc9bba20080fbb3 2023-04-16 00:56:30,557 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:30,557 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:30,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:30,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:30,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:30,557 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:30,557 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:56:30,559 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/27642fd671a84c598fc9bba20080fbb3, entries=150, sequenceid=1098, filesize=11.9 K 2023-04-16 00:56:30,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:30,562 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=134.18 KB/137400 for d538367277459b70f4a06235dfaacfec in 1262ms, sequenceid=1098, compaction requested=true 2023-04-16 00:56:30,562 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for d538367277459b70f4a06235dfaacfec: 2023-04-16 00:56:30,562 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 00:56:30,562 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-16 00:56:30,562 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-16 00:56:30,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:30,562 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 00:56:30,563 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 00:56:30,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:30,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:30,564 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 107430 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-16 00:56:30,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:30,564 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HStore(1912): d538367277459b70f4a06235dfaacfec/A is initiating minor compaction (all files) 2023-04-16 00:56:30,564 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of d538367277459b70f4a06235dfaacfec/A in TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. 2023-04-16 00:56:30,564 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A/1122a71abf6a45af95f9c3a6185f56fe, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A/a5afc425ea0b4a36acfd05a67e740914, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A/d97367c08cde484ba1badf53f5a0366f] into tmpdir=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp, totalSize=104.9 K 2023-04-16 00:56:30,564 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37716 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-16 00:56:30,564 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HStore(1912): d538367277459b70f4a06235dfaacfec/B is initiating minor compaction (all files) 2023-04-16 00:56:30,564 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:56:30,564 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HRegion(2259): Starting compaction of d538367277459b70f4a06235dfaacfec/B in TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. 2023-04-16 00:56:30,564 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:56:30,564 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:30,564 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@21d2543c[Count = 0] remaining members to acquire global barrier 2023-04-16 00:56:30,564 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/e21e02618d174623b613a8d1163f365f, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/1b14d0512fb54b2ca3808faeebf965a6, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/c95993956ef44794ab016c69f0c94946] into tmpdir=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp, totalSize=36.8 K 2023-04-16 00:56:30,565 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.Compactor(207): Compacting 1122a71abf6a45af95f9c3a6185f56fe, keycount=150, bloomtype=ROW, size=26.5 K, encoding=NONE, compression=NONE, seqNum=1066, earliestPutTs=1681606587980 2023-04-16 00:56:30,565 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting e21e02618d174623b613a8d1163f365f, keycount=150, bloomtype=ROW, size=13.0 K, encoding=NONE, compression=NONE, seqNum=1066, earliestPutTs=1681606587980 2023-04-16 00:56:30,565 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.Compactor(207): Compacting a5afc425ea0b4a36acfd05a67e740914, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=1086, earliestPutTs=1681606588149 2023-04-16 00:56:30,565 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting 1b14d0512fb54b2ca3808faeebf965a6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1086, earliestPutTs=1681606588149 2023-04-16 00:56:30,565 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:30,565 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.Compactor(207): Compacting d97367c08cde484ba1badf53f5a0366f, keycount=350, bloomtype=ROW, size=52.9 K, encoding=NONE, compression=NONE, seqNum=1098, earliestPutTs=1681606589289 2023-04-16 00:56:30,565 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:30,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:30,566 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:56:30,566 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:56:30,566 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:56:30,566 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:30,566 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:56:30,566 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting c95993956ef44794ab016c69f0c94946, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1098, earliestPutTs=1681606589299 2023-04-16 00:56:30,568 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:30,568 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:56:30,568 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:56:30,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:30,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:30,568 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:56:30,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:30,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:30,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:30,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:30,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:30,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:30,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:30,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:30,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:56:30,570 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:56:30,570 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:56:30,570 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:56:30,570 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:56:30,570 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:56:30,570 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:56:30,572 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:30,572 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:30,572 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:30,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:30,572 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:56:30,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:30,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:30,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:30,572 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:56:30,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:56:30,573 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:30,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:30,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:30,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:30,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:30,574 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] throttle.PressureAwareThroughputController(145): d538367277459b70f4a06235dfaacfec#B#compaction#666 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-16 00:56:30,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:30,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:30,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:30,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:30,576 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] throttle.PressureAwareThroughputController(145): d538367277459b70f4a06235dfaacfec#A#compaction#665 average throughput is 4.93 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-16 00:56:30,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:30,577 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:30,579 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:30,579 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:30,579 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:56:30,579 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:56:30,579 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:56:30,579 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:56:30,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:56:30,579 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:56:30,579 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:56:30,579 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:30,579 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:30,579 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:30,579 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:30,579 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:30,579 DEBUG [Thread-4138] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:56:30,579 DEBUG [Thread-4138] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:56:30,580 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:56:30,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:56:30,608 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp/A/67fe03bbff1c41b6adeccd6cee68c9e2 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A/67fe03bbff1c41b6adeccd6cee68c9e2 2023-04-16 00:56:30,611 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in d538367277459b70f4a06235dfaacfec/A of d538367277459b70f4a06235dfaacfec into 67fe03bbff1c41b6adeccd6cee68c9e2(size=26.6 K), total size for store is 26.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 00:56:30,611 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for d538367277459b70f4a06235dfaacfec: 2023-04-16 00:56:30,611 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec., storeName=d538367277459b70f4a06235dfaacfec/A, priority=13, startTime=1681606590562; duration=0sec 2023-04-16 00:56:30,611 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 00:56:30,611 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-16 00:56:30,612 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37716 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-16 00:56:30,612 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HStore(1912): d538367277459b70f4a06235dfaacfec/C is initiating minor compaction (all files) 2023-04-16 00:56:30,612 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of d538367277459b70f4a06235dfaacfec/C in TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. 2023-04-16 00:56:30,612 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/1ca76d69bd8841bd947f25033ae474d3, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/e4ed3fb509e94ee294acb2f9a36aada0, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/27642fd671a84c598fc9bba20080fbb3] into tmpdir=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp, totalSize=36.8 K 2023-04-16 00:56:30,613 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.Compactor(207): Compacting 1ca76d69bd8841bd947f25033ae474d3, keycount=150, bloomtype=ROW, size=13.0 K, encoding=NONE, compression=NONE, seqNum=1066, earliestPutTs=1681606587980 2023-04-16 00:56:30,613 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.Compactor(207): Compacting e4ed3fb509e94ee294acb2f9a36aada0, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1086, earliestPutTs=1681606588149 2023-04-16 00:56:30,613 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.Compactor(207): Compacting 27642fd671a84c598fc9bba20080fbb3, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1098, earliestPutTs=1681606589299 2023-04-16 00:56:30,618 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] throttle.PressureAwareThroughputController(145): d538367277459b70f4a06235dfaacfec#C#compaction#667 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-16 00:56:30,631 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp/C/2bbb9b6739aa4649a9850bd0ce9ac919 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/2bbb9b6739aa4649a9850bd0ce9ac919 2023-04-16 00:56:30,634 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in d538367277459b70f4a06235dfaacfec/C of d538367277459b70f4a06235dfaacfec into 2bbb9b6739aa4649a9850bd0ce9ac919(size=13.1 K), total size for store is 13.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 00:56:30,635 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for d538367277459b70f4a06235dfaacfec: 2023-04-16 00:56:30,635 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec., storeName=d538367277459b70f4a06235dfaacfec/C, priority=13, startTime=1681606590563; duration=0sec 2023-04-16 00:56:30,635 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 00:56:30,679 DEBUG [Thread-4138] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:56:30,680 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:56:30,682 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:56:30,683 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:56:30,683 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:56:30,683 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:56:30,683 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:56:30,683 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:56:30,683 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:56:30,683 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:30,683 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:30,688 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:30,688 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:56:30,688 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:56:30,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:56:30,688 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:30,689 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:56:30,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:30,689 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:30,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:56:30,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:30,689 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:56:30,690 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:56:30,690 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:56:30,690 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:56:30,690 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:56:30,690 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:56:30,690 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:56:30,690 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. 2023-04-16 00:56:30,690 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. started... 2023-04-16 00:56:30,690 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:56:30,690 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing d538367277459b70f4a06235dfaacfec 3/3 column families, dataSize=134.18 KB heapSize=380.30 KB 2023-04-16 00:56:30,690 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d538367277459b70f4a06235dfaacfec, store=A 2023-04-16 00:56:30,690 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 00:56:30,690 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d538367277459b70f4a06235dfaacfec, store=B 2023-04-16 00:56:30,690 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 00:56:30,691 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d538367277459b70f4a06235dfaacfec, store=C 2023-04-16 00:56:30,691 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 00:56:30,720 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304160e39d029a4b54df59cbfbbc7420a3542 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304160e39d029a4b54df59cbfbbc7420a3542 2023-04-16 00:56:30,726 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=1136, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp/A/b18b0a2fbb8640388db4ee137f17466e 2023-04-16 00:56:30,736 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=1136 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp/B/007c353b19c74958a6b7f5cc4c51fece 2023-04-16 00:56:30,748 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=1136 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp/C/0cfe7ca6635a4b31a3eabc5368f5aec1 2023-04-16 00:56:30,751 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp/A/b18b0a2fbb8640388db4ee137f17466e as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A/b18b0a2fbb8640388db4ee137f17466e 2023-04-16 00:56:30,755 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A/b18b0a2fbb8640388db4ee137f17466e, entries=150, sequenceid=1136, filesize=25.4 K 2023-04-16 00:56:30,755 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp/B/007c353b19c74958a6b7f5cc4c51fece as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/007c353b19c74958a6b7f5cc4c51fece 2023-04-16 00:56:30,758 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/007c353b19c74958a6b7f5cc4c51fece, entries=150, sequenceid=1136, filesize=11.9 K 2023-04-16 00:56:30,759 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp/C/0cfe7ca6635a4b31a3eabc5368f5aec1 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/0cfe7ca6635a4b31a3eabc5368f5aec1 2023-04-16 00:56:30,762 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/0cfe7ca6635a4b31a3eabc5368f5aec1, entries=150, sequenceid=1136, filesize=11.9 K 2023-04-16 00:56:30,763 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~380.25 KB/389376, currentSize=0 B/0 for d538367277459b70f4a06235dfaacfec in 73ms, sequenceid=1136, compaction requested=false 2023-04-16 00:56:30,763 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for d538367277459b70f4a06235dfaacfec: 2023-04-16 00:56:30,763 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. 2023-04-16 00:56:30,763 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:56:30,763 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:56:30,763 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:30,763 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:56:30,763 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:56:30,764 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:30,764 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:30,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:30,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:30,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:30,765 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:30,765 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:56:30,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:30,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:30,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:30,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:30,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:30,767 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:56:30,767 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@760e605d[Count = 0] remaining members to acquire global barrier 2023-04-16 00:56:30,767 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:56:30,767 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:30,768 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:30,768 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:30,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:30,768 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:56:30,768 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:56:30,768 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:56:30,768 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:30,768 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:56:30,769 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:56:30,769 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:56:30,769 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:30,770 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:56:30,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:30,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:30,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:30,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:30,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:30,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:30,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:30,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:30,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:30,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:30,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:56:30,772 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:56:30,772 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:56:30,772 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:56:30,772 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:56:30,772 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:56:30,772 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:56:30,774 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:30,775 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:56:30,775 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:30,775 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:30,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:30,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:30,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:30,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:30,775 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:56:30,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:56:30,775 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:30,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:30,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:30,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:30,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:30,776 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:30,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:30,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:30,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:30,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:30,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:30,779 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:30,779 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:30,779 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:56:30,779 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:56:30,779 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:56:30,779 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:56:30,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:56:30,779 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:56:30,779 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:56:30,779 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:30,779 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:30,779 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:30,779 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:56:30,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:56:30,779 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:30,779 DEBUG [Thread-4138] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:56:30,779 DEBUG [Thread-4138] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:56:30,779 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:30,780 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:56:30,879 DEBUG [Thread-4138] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:56:30,880 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:56:30,882 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:56:30,883 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:56:30,883 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:56:30,883 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:56:30,883 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:56:30,883 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:56:30,883 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:56:30,883 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:30,884 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:30,885 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:56:30,885 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:30,885 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:56:30,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:56:30,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:30,885 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:30,885 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:56:30,886 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:30,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:56:30,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:30,886 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:56:30,886 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:56:30,886 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:56:30,886 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:56:30,886 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:56:30,886 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:56:30,887 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:56:30,887 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. 2023-04-16 00:56:30,887 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. started... 2023-04-16 00:56:30,887 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:56:30,887 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for d538367277459b70f4a06235dfaacfec: 2023-04-16 00:56:30,887 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. 2023-04-16 00:56:30,887 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:56:30,887 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:56:30,887 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:30,887 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:56:30,887 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:56:30,889 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:30,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:30,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:30,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:30,889 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:30,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:30,889 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:30,889 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:56:30,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:30,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:30,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:30,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:30,890 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:56:30,891 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@65b8f659[Count = 0] remaining members to acquire global barrier 2023-04-16 00:56:30,891 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:56:30,891 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:30,893 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:30,893 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:30,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:30,893 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:56:30,893 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:56:30,893 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:56:30,893 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:30,893 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:56:30,894 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:30,894 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:56:30,895 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:56:30,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:30,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:30,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:30,895 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:56:30,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:30,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:30,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:30,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:30,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:30,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:30,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:30,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:56:30,897 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:56:30,897 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:56:30,898 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:56:30,898 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:56:30,898 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:56:30,898 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:56:30,899 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:30,899 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:56:30,899 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:30,899 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:30,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:30,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:30,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:30,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:30,900 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:56:30,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:56:30,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:30,900 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:30,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:30,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:30,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:30,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:30,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:30,901 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:30,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:30,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:30,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:30,903 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:56:30,903 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:56:30,903 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:30,903 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:30,904 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:56:30,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:56:30,904 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:30,904 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:30,904 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:30,904 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:30,904 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:30,904 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:56:30,904 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:56:30,904 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:56:30,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:56:30,904 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:56:30,904 DEBUG [Thread-4138] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:56:30,904 DEBUG [Thread-4138] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:56:30,993 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp/B/2e98666f01c24e69a099e8eee156ee6e as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/2e98666f01c24e69a099e8eee156ee6e 2023-04-16 00:56:30,997 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in d538367277459b70f4a06235dfaacfec/B of d538367277459b70f4a06235dfaacfec into 2e98666f01c24e69a099e8eee156ee6e(size=13.1 K), total size for store is 25.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 00:56:30,997 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for d538367277459b70f4a06235dfaacfec: 2023-04-16 00:56:30,997 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec., storeName=d538367277459b70f4a06235dfaacfec/B, priority=13, startTime=1681606590562; duration=0sec 2023-04-16 00:56:30,997 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 00:56:31,004 DEBUG [Thread-4138] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:56:31,005 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:56:31,007 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:56:31,008 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:56:31,008 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:56:31,008 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:56:31,008 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:56:31,008 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:56:31,008 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:56:31,008 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:31,008 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:31,010 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:31,010 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:56:31,010 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:56:31,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:56:31,011 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:31,011 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:56:31,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:31,011 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:31,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:56:31,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:31,011 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:56:31,012 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:56:31,012 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:56:31,012 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:56:31,012 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:56:31,012 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:56:31,012 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:56:31,012 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:56:31,012 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. 2023-04-16 00:56:31,012 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. started... 2023-04-16 00:56:31,013 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for d538367277459b70f4a06235dfaacfec: 2023-04-16 00:56:31,013 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. 2023-04-16 00:56:31,013 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:56:31,013 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:56:31,013 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:31,013 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:56:31,013 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:56:31,014 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:31,014 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:31,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:31,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:31,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:31,015 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:31,015 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:56:31,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:31,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:31,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:31,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:31,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:31,016 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:56:31,016 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@595c5d9c[Count = 0] remaining members to acquire global barrier 2023-04-16 00:56:31,016 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:56:31,016 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:31,018 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:31,018 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:31,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:31,018 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:56:31,018 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:56:31,018 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:56:31,018 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:31,018 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:56:31,019 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:56:31,019 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:31,019 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:56:31,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:31,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:31,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:31,019 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:56:31,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:31,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:31,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:31,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:31,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:31,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:31,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:31,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:56:31,022 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:56:31,022 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:56:31,022 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:56:31,022 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:56:31,022 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:56:31,022 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:56:31,024 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:31,024 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:31,024 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:31,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:31,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:31,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:31,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:31,024 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:56:31,025 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:56:31,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:56:31,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:31,025 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:31,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:31,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:31,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:31,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:31,026 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:31,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:31,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:31,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:31,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:31,028 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:31,028 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:31,028 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:31,028 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:56:31,028 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:56:31,028 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:56:31,028 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:56:31,028 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:56:31,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:56:31,028 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:56:31,028 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:31,029 DEBUG [Thread-4138] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:56:31,029 DEBUG [Thread-4138] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:56:31,028 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:56:31,029 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:31,029 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:56:31,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:56:31,029 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:31,030 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:31,129 DEBUG [Thread-4138] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:56:31,129 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:56:31,131 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:56:31,132 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:56:31,132 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:56:31,132 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:56:31,132 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:56:31,132 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:56:31,132 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:56:31,133 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:31,133 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:31,134 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:31,134 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:56:31,134 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:56:31,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:56:31,134 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:31,134 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:56:31,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:31,135 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:31,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:56:31,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:31,135 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:56:31,135 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:56:31,135 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:56:31,136 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:56:31,136 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:56:31,136 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:56:31,136 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:56:31,136 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. 2023-04-16 00:56:31,136 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. started... 2023-04-16 00:56:31,136 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:56:31,136 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for d538367277459b70f4a06235dfaacfec: 2023-04-16 00:56:31,136 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. 2023-04-16 00:56:31,136 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:56:31,136 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:56:31,136 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:31,136 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:56:31,136 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:56:31,141 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:31,141 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:31,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:31,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:31,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:31,141 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:31,141 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:56:31,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:31,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:31,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:31,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:31,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:31,143 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:56:31,143 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6a1c913f[Count = 0] remaining members to acquire global barrier 2023-04-16 00:56:31,143 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:56:31,143 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:31,144 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:31,144 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:31,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:31,144 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:56:31,144 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:56:31,144 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:31,144 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:56:31,144 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:56:31,145 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:31,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:31,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:31,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:31,145 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:56:31,146 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:56:31,146 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:56:31,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:31,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:31,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:31,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:31,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:31,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:31,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:31,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:56:31,148 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:56:31,148 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:56:31,148 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:56:31,148 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:56:31,148 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:56:31,148 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:56:31,150 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:31,150 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:31,150 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:31,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:31,150 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:56:31,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:31,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:31,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:31,150 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:56:31,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:56:31,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:31,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:31,150 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:31,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:31,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:31,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:31,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:31,152 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:31,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:31,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:31,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:31,155 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:56:31,155 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:56:31,155 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:31,155 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:56:31,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:56:31,156 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:31,156 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:31,156 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:31,156 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:31,156 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:31,156 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:31,156 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:56:31,156 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:56:31,155 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:56:31,156 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:56:31,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:56:31,156 DEBUG [Thread-4138] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:56:31,156 DEBUG [Thread-4138] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:56:31,256 DEBUG [Thread-4138] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:56:31,257 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:56:31,258 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:56:31,259 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:56:31,259 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:56:31,260 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:56:31,260 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:56:31,260 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:56:31,260 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:56:31,260 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:31,260 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:31,261 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:56:31,261 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:31,261 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:56:31,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:56:31,262 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:31,262 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:56:31,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:31,262 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:31,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:56:31,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:31,262 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:56:31,263 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:56:31,263 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:56:31,263 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:56:31,263 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:56:31,263 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:56:31,263 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. 2023-04-16 00:56:31,263 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. started... 2023-04-16 00:56:31,263 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:56:31,263 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for d538367277459b70f4a06235dfaacfec: 2023-04-16 00:56:31,263 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. 2023-04-16 00:56:31,263 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:56:31,263 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:56:31,263 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:31,263 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:56:31,263 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:56:31,265 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:31,265 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:31,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:31,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:31,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:31,265 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:31,265 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:56:31,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:31,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:31,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:31,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:31,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:31,267 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:56:31,267 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6f9518a9[Count = 0] remaining members to acquire global barrier 2023-04-16 00:56:31,267 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:56:31,267 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:31,268 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:31,268 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:31,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:31,268 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:56:31,268 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:56:31,268 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:56:31,268 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:31,268 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:56:31,271 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:31,271 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:56:31,271 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:56:31,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:31,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:31,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:31,271 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:56:31,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:31,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:31,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:31,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:31,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:31,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:31,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:31,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:56:31,273 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:56:31,273 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:56:31,274 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:56:31,274 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:56:31,274 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:56:31,274 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:56:31,275 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:31,275 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:56:31,275 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:31,275 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:31,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:31,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:31,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:31,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:31,276 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:56:31,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:56:31,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:31,276 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:31,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:31,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:31,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:31,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:31,277 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:31,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:31,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:31,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:31,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:31,279 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:31,279 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:56:31,279 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:56:31,279 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:31,279 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:56:31,279 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:56:31,279 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:31,279 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:31,279 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:31,279 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:56:31,279 DEBUG [Thread-4138] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:56:31,279 DEBUG [Thread-4138] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:56:31,279 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:56:31,279 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:56:31,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:56:31,279 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:31,280 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:31,280 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:56:31,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:56:31,379 DEBUG [Thread-4138] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:56:31,380 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:56:31,382 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:56:31,383 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:56:31,383 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:56:31,383 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:56:31,383 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:56:31,383 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:56:31,383 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:56:31,383 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:31,383 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:31,384 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:56:31,384 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:31,384 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:56:31,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:56:31,385 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:31,385 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:56:31,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:31,385 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:31,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:56:31,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:31,386 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:56:31,386 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:56:31,386 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:56:31,386 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:56:31,386 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:56:31,386 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:56:31,386 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:56:31,386 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. 2023-04-16 00:56:31,386 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. started... 2023-04-16 00:56:31,386 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:56:31,386 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for d538367277459b70f4a06235dfaacfec: 2023-04-16 00:56:31,386 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. 2023-04-16 00:56:31,386 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:56:31,386 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:56:31,387 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:31,387 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:56:31,387 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:56:31,389 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:31,389 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:31,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:31,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:31,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:31,389 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:31,389 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:56:31,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:31,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:31,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:31,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:31,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:31,391 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:56:31,391 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4d3d313c[Count = 0] remaining members to acquire global barrier 2023-04-16 00:56:31,391 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:56:31,391 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:31,392 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:31,392 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:31,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:31,392 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:56:31,392 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:56:31,392 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:56:31,392 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:31,392 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:56:31,393 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:31,393 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:56:31,393 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:56:31,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:31,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:31,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:31,393 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:56:31,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:31,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:31,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:31,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:31,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:31,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:31,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:31,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:56:31,395 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:56:31,395 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:56:31,395 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:56:31,395 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:56:31,396 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:56:31,396 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:56:31,397 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:31,397 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:31,397 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:56:31,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:31,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:31,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:31,397 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:31,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:31,397 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:31,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:31,398 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:56:31,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:56:31,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:31,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:31,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:31,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:31,399 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:31,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:31,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:31,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:31,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:31,403 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:31,403 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:31,403 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:31,403 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:31,403 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:31,403 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:31,403 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:31,403 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:56:31,403 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:56:31,403 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:56:31,404 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:56:31,404 DEBUG [Thread-4138] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:56:31,404 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:56:31,404 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:56:31,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:56:31,404 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:56:31,404 DEBUG [Thread-4138] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:56:31,405 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:56:31,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:56:31,473 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(9158): Flush requested on d538367277459b70f4a06235dfaacfec 2023-04-16 00:56:31,473 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing d538367277459b70f4a06235dfaacfec 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-16 00:56:31,474 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d538367277459b70f4a06235dfaacfec, store=A 2023-04-16 00:56:31,474 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:56:31,474 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d538367277459b70f4a06235dfaacfec, store=B 2023-04-16 00:56:31,474 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:56:31,474 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d538367277459b70f4a06235dfaacfec, store=C 2023-04-16 00:56:31,474 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:56:31,501 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681606591486, max timestamp=1681606591499Num uniques 859; 2023-04-16 00:56:31,501 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-16 00:56:31,501 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681606591486, max timestamp=1681606591499Num uniques 859; 2023-04-16 00:56:31,501 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-16 00:56:31,501 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681606591486, max timestamp=1681606591499Num uniques 859; 2023-04-16 00:56:31,501 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-16 00:56:31,504 DEBUG [Thread-4138] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:56:31,505 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:56:31,506 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:56:31,507 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:56:31,507 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:56:31,507 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:56:31,507 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:56:31,507 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:56:31,507 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:56:31,508 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:31,508 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:31,509 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:56:31,509 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:56:31,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:56:31,509 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:31,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:31,510 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:31,510 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:56:31,510 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:31,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:56:31,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:31,510 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:56:31,511 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:56:31,511 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:56:31,511 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:56:31,511 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:56:31,511 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:56:31,511 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:56:31,511 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:56:31,511 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. 2023-04-16 00:56:31,511 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. started... 2023-04-16 00:56:31,511 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. as already flushing 2023-04-16 00:56:31,511 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. 2023-04-16 00:56:31,511 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:56:31,511 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:56:31,511 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:31,511 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:56:31,511 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:56:31,513 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:31,513 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:31,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:31,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:31,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:31,514 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:31,514 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:56:31,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:31,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:31,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:31,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:31,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:31,515 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:56:31,515 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@763635ba[Count = 0] remaining members to acquire global barrier 2023-04-16 00:56:31,515 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:56:31,515 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:31,516 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:31,516 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:31,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:31,516 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:56:31,516 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:56:31,516 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:56:31,517 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:31,517 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:56:31,518 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:31,518 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 365 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44548 deadline: 1681606651513, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:31,521 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:56:31,521 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:31,521 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:31,521 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 330 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44554 deadline: 1681606651516, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:31,521 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:56:31,521 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:56:31,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:31,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:31,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:31,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:31,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:31,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:31,522 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:31,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:31,523 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 300 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44562 deadline: 1681606651518, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:31,523 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:31,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:31,523 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 257 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44564 deadline: 1681606651519, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:31,523 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:31,523 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 276 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44556 deadline: 1681606651519, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:31,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:31,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:31,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:56:31,525 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:56:31,525 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:56:31,525 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:56:31,525 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:56:31,525 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:56:31,525 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:56:31,527 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:31,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:31,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:31,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:31,527 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:31,527 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:56:31,527 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:31,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:31,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:31,527 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:31,527 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:56:31,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:56:31,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:31,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:31,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:31,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:31,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:31,529 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:31,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:31,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:31,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:31,531 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:31,531 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:56:31,531 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:56:31,531 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:56:31,531 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:56:31,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:56:31,531 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:56:31,531 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:56:31,531 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:31,531 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:56:31,531 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:31,532 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:31,532 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:56:31,532 DEBUG [Thread-4138] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:56:31,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:56:31,532 DEBUG [Thread-4138] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:56:31,532 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:31,532 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:31,532 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:31,621 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:31,621 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 367 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44548 deadline: 1681606651619, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:31,624 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:31,624 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 332 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44554 deadline: 1681606651622, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:31,626 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:31,626 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 302 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44562 deadline: 1681606651624, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:31,626 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:31,626 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 259 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44564 deadline: 1681606651624, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:31,626 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:31,626 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 278 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44556 deadline: 1681606651624, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:31,632 DEBUG [Thread-4138] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:56:31,632 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:56:31,634 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:56:31,635 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:56:31,635 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:56:31,635 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:56:31,635 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:56:31,635 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:56:31,635 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:56:31,636 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:31,636 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:31,637 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:31,637 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:56:31,637 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:56:31,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:56:31,637 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:31,637 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:56:31,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:31,638 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:31,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:56:31,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:31,638 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:56:31,638 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:56:31,639 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:56:31,639 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:56:31,639 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:56:31,639 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:56:31,639 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:56:31,639 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. 2023-04-16 00:56:31,639 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:56:31,639 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. started... 2023-04-16 00:56:31,639 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. as already flushing 2023-04-16 00:56:31,639 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. 2023-04-16 00:56:31,639 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:56:31,640 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:56:31,640 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:31,640 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:56:31,640 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:56:31,642 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:31,642 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:31,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:31,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:31,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:31,642 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:31,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:31,643 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:56:31,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:31,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:31,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:31,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:31,644 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:56:31,645 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@782dd72a[Count = 0] remaining members to acquire global barrier 2023-04-16 00:56:31,645 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:56:31,645 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:31,646 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:31,646 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:31,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:31,646 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:56:31,646 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:56:31,646 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:56:31,646 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:31,646 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:56:31,647 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:56:31,647 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:56:31,647 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:31,647 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:56:31,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:31,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:31,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:31,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:31,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:31,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:31,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:31,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:31,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:31,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:31,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:56:31,650 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:56:31,650 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:56:31,650 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:56:31,650 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:56:31,650 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:56:31,650 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:56:31,651 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:31,651 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:31,652 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:56:31,652 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:31,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:31,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:31,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:31,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:31,652 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:56:31,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:56:31,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:31,652 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:31,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:31,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:31,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:31,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:31,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:31,654 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:31,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:31,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:31,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:31,657 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:56:31,657 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:56:31,657 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:56:31,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:56:31,657 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:56:31,657 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:56:31,657 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:56:31,657 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:31,658 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:31,658 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:31,658 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:31,658 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:31,658 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:31,658 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:31,658 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:56:31,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:56:31,658 DEBUG [Thread-4138] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:56:31,658 DEBUG [Thread-4138] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:56:31,758 DEBUG [Thread-4138] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:56:31,759 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:56:31,760 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:56:31,761 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:56:31,761 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:56:31,761 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:56:31,762 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:56:31,762 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:56:31,762 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:56:31,762 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:31,762 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:31,763 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:56:31,763 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:31,763 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:56:31,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:56:31,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:31,764 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:31,764 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:56:31,764 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:31,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:56:31,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:31,764 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:56:31,765 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:56:31,765 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:56:31,765 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:56:31,765 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:56:31,765 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:56:31,765 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:56:31,765 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:56:31,765 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. 2023-04-16 00:56:31,765 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. started... 2023-04-16 00:56:31,765 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. as already flushing 2023-04-16 00:56:31,766 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. 2023-04-16 00:56:31,766 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:56:31,766 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:56:31,766 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:31,766 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:56:31,766 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:56:31,767 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:31,767 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:31,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:31,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:31,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:31,768 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:31,768 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:56:31,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:31,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:31,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:31,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:31,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:31,769 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:56:31,769 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3f3ad8d6[Count = 0] remaining members to acquire global barrier 2023-04-16 00:56:31,769 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:56:31,769 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:31,770 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:31,770 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:31,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:31,771 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:56:31,771 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:56:31,771 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:56:31,771 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:31,771 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:56:31,773 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:31,773 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:56:31,773 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:56:31,773 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:56:31,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:31,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:31,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:31,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:31,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:31,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:31,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:31,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:31,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:31,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:31,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:56:31,775 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:56:31,775 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:56:31,775 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:56:31,775 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:56:31,775 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:56:31,775 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:56:31,777 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:31,777 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:31,777 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:56:31,777 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:31,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:31,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:31,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:31,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:31,777 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:56:31,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:56:31,777 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:31,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:31,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:31,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:31,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:31,778 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:31,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:31,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:31,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:31,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:31,781 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:31,781 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:31,781 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:31,781 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:31,781 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:31,781 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:56:31,781 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:56:31,781 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:31,781 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:31,781 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:56:31,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:56:31,781 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:56:31,781 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:56:31,781 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:56:31,781 DEBUG [Thread-4138] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:56:31,781 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:56:31,781 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:56:31,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:56:31,781 DEBUG [Thread-4138] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:56:31,825 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:31,825 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 369 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44548 deadline: 1681606651823, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:31,828 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:31,828 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 334 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44554 deadline: 1681606651826, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:31,828 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:31,829 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 304 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44562 deadline: 1681606651827, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:31,829 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:31,829 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 280 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44556 deadline: 1681606651827, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:31,829 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:31,829 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 261 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44564 deadline: 1681606651828, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:31,882 DEBUG [Thread-4138] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:56:31,882 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:56:31,884 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:56:31,885 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:56:31,885 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:56:31,885 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:56:31,885 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:56:31,885 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:56:31,886 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:56:31,886 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:31,886 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:31,887 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:56:31,887 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:56:31,887 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:31,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:56:31,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:31,887 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:31,888 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:56:31,888 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:31,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:56:31,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:31,888 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:56:31,888 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:56:31,889 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:56:31,889 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:56:31,889 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:56:31,889 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:56:31,889 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:56:31,889 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:56:31,889 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. 2023-04-16 00:56:31,889 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. started... 2023-04-16 00:56:31,889 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. as already flushing 2023-04-16 00:56:31,889 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. 2023-04-16 00:56:31,889 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:56:31,889 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:56:31,889 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:31,889 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:56:31,889 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:56:31,892 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:31,892 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:31,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:31,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:31,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:31,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:31,893 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:31,893 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:56:31,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:31,893 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304167c01a5c4663740dab31544804960d89f to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304167c01a5c4663740dab31544804960d89f 2023-04-16 00:56:31,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:31,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:31,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:31,894 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:56:31,894 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1b2abadc[Count = 0] remaining members to acquire global barrier 2023-04-16 00:56:31,894 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:56:31,894 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:31,895 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:31,895 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:31,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:31,896 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:56:31,896 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:56:31,896 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:31,896 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:56:31,896 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:56:31,897 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:31,897 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:56:31,897 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:56:31,897 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:56:31,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:31,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:31,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:31,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:31,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:31,898 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=1153, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp/A/682a8b9d78f9467d8142c539b349897e 2023-04-16 00:56:31,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:31,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:31,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:31,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:31,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:31,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:56:31,900 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:56:31,900 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:56:31,900 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:56:31,900 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:56:31,900 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:56:31,900 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:56:31,906 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:31,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:31,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:31,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:31,906 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:31,906 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:56:31,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:31,906 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:31,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:31,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:31,907 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:56:31,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:56:31,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:31,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:31,908 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:31,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:31,909 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=1153 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp/B/703ddd1cbced43b287286c523fcea3fa 2023-04-16 00:56:31,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:31,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:31,910 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:31,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:31,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:31,914 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:56:31,914 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:56:31,914 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:56:31,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:56:31,914 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:31,914 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:31,914 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:31,914 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:31,915 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:31,915 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:31,915 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:31,915 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:56:31,915 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:56:31,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:56:31,916 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:56:31,916 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:56:31,916 DEBUG [Thread-4138] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:56:31,916 DEBUG [Thread-4138] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:56:32,016 DEBUG [Thread-4138] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:56:32,017 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:56:32,018 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:56:32,020 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:56:32,020 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:56:32,020 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:56:32,020 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:56:32,020 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:56:32,020 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:56:32,020 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:32,020 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:32,022 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:32,022 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:56:32,022 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:56:32,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:56:32,022 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:32,022 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:56:32,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:32,022 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:32,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:56:32,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:32,023 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:56:32,023 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:56:32,023 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:56:32,023 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:56:32,023 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:56:32,023 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:56:32,024 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:56:32,024 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:56:32,024 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. 2023-04-16 00:56:32,024 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. started... 2023-04-16 00:56:32,024 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. as already flushing 2023-04-16 00:56:32,024 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. 2023-04-16 00:56:32,024 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:56:32,024 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:56:32,024 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:32,024 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:56:32,024 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:56:32,026 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:32,026 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:32,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:32,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:32,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:32,026 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:32,026 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:56:32,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:32,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:32,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:32,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:32,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:32,027 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:56:32,027 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5cf6d7e6[Count = 0] remaining members to acquire global barrier 2023-04-16 00:56:32,027 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:56:32,027 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:32,030 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:32,030 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:32,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:32,030 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:56:32,030 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:56:32,030 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:56:32,030 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:32,030 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:56:32,031 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:32,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:32,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:32,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:32,032 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:56:32,032 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:56:32,032 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:56:32,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:32,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:32,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:32,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:32,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:32,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:32,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:32,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:56:32,034 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:56:32,034 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:56:32,034 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:56:32,034 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:56:32,034 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:56:32,034 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:56:32,036 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:32,036 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:32,036 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:56:32,036 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:32,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:32,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:32,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:32,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:32,036 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:56:32,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:56:32,036 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:32,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:32,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:32,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:32,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:32,038 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:32,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:32,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:32,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:32,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:32,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:32,041 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:32,041 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:56:32,041 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:32,041 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:56:32,041 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:56:32,041 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:56:32,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:56:32,041 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:56:32,041 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:56:32,041 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:32,041 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:56:32,041 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:32,041 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:56:32,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:56:32,042 DEBUG [Thread-4138] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:56:32,042 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:32,042 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:32,042 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:32,042 DEBUG [Thread-4138] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:56:32,128 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:32,129 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 371 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44548 deadline: 1681606652127, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:32,132 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:32,132 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 336 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44554 deadline: 1681606652129, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:32,132 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:32,132 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 282 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44556 deadline: 1681606652131, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:32,132 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:32,133 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 263 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44564 deadline: 1681606652131, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:32,134 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:32,134 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 306 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44562 deadline: 1681606652131, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:32,142 DEBUG [Thread-4138] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:56:32,142 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:56:32,144 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:56:32,145 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:56:32,145 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:56:32,145 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:56:32,145 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:56:32,145 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:56:32,145 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:56:32,145 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:32,145 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:32,147 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:56:32,147 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:56:32,147 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:32,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:56:32,148 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:32,148 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:56:32,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:32,148 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:32,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:56:32,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:32,148 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:56:32,148 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:56:32,149 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:56:32,149 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:56:32,149 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:56:32,149 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:56:32,149 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:56:32,149 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:56:32,149 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. 2023-04-16 00:56:32,149 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. started... 2023-04-16 00:56:32,149 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. as already flushing 2023-04-16 00:56:32,149 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. 2023-04-16 00:56:32,149 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:56:32,149 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:56:32,149 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:32,149 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:56:32,149 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:56:32,151 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:32,151 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:32,151 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:56:32,151 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:32,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:32,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:32,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:32,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:32,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:32,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:32,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:32,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:32,152 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:56:32,153 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4a74dfd5[Count = 0] remaining members to acquire global barrier 2023-04-16 00:56:32,153 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:56:32,153 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:32,154 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:32,154 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:32,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:32,154 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:56:32,154 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:56:32,154 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:32,154 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:56:32,154 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:56:32,155 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:56:32,155 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:32,155 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:56:32,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:32,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:32,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:32,155 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:56:32,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:32,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:32,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:32,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:32,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:32,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:32,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:32,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:56:32,157 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:56:32,157 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:56:32,157 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:56:32,158 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:56:32,158 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:56:32,158 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:56:32,159 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:32,160 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:32,160 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:56:32,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:32,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:32,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:32,160 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:32,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:32,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:32,160 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:56:32,160 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:32,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:56:32,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:32,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:32,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:32,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:32,161 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:32,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:32,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:32,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:32,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:32,163 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:56:32,163 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:32,164 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:32,164 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:32,164 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:32,164 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:32,164 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:32,163 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:56:32,163 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:56:32,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:56:32,164 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:32,164 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:56:32,164 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:56:32,164 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:56:32,164 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:56:32,164 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:56:32,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:56:32,164 DEBUG [Thread-4138] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:56:32,164 DEBUG [Thread-4138] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:56:32,264 DEBUG [Thread-4138] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:56:32,265 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:56:32,267 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:56:32,268 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:56:32,268 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:56:32,268 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:56:32,268 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:56:32,268 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:56:32,268 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:56:32,268 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:32,268 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:32,269 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:56:32,269 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:32,269 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:56:32,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:56:32,270 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:32,270 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:56:32,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:32,270 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:32,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:56:32,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:32,270 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:56:32,271 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:56:32,271 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:56:32,271 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:56:32,271 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:56:32,271 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:56:32,271 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:56:32,271 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:56:32,271 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. 2023-04-16 00:56:32,271 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. started... 2023-04-16 00:56:32,271 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. as already flushing 2023-04-16 00:56:32,271 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. 2023-04-16 00:56:32,271 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:56:32,271 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:56:32,271 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:32,271 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:56:32,271 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:56:32,273 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:32,273 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:32,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:32,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:32,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:32,273 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:32,273 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:56:32,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:32,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:32,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:32,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:32,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:32,274 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:56:32,274 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@23615225[Count = 0] remaining members to acquire global barrier 2023-04-16 00:56:32,274 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:56:32,274 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:32,276 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:32,277 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:32,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:32,277 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:32,277 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:56:32,277 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:56:32,277 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:56:32,277 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:56:32,278 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:32,278 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:56:32,278 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:56:32,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:32,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:32,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:32,278 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:56:32,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:32,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:32,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:32,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:32,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:32,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:32,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:32,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:56:32,280 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:56:32,280 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:56:32,280 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:56:32,280 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:56:32,280 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:56:32,280 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:56:32,282 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:32,282 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:32,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:32,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:32,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:32,282 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:32,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:32,282 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:56:32,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:32,282 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:56:32,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:56:32,283 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:32,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:32,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:32,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:32,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:32,284 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:32,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:32,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:32,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:32,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:32,289 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:56:32,289 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:56:32,289 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:56:32,289 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:56:32,289 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:56:32,289 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:32,289 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:56:32,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:56:32,289 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:32,289 DEBUG [Thread-4138] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:56:32,289 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:32,290 DEBUG [Thread-4138] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:56:32,290 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:32,290 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:56:32,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:56:32,290 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:32,290 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:32,290 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:32,328 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=1153 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp/C/999304bf7b3e4156b92df1e201a7bffc 2023-04-16 00:56:32,332 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp/A/682a8b9d78f9467d8142c539b349897e as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A/682a8b9d78f9467d8142c539b349897e 2023-04-16 00:56:32,335 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A/682a8b9d78f9467d8142c539b349897e, entries=400, sequenceid=1153, filesize=59.8 K 2023-04-16 00:56:32,336 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp/B/703ddd1cbced43b287286c523fcea3fa as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/703ddd1cbced43b287286c523fcea3fa 2023-04-16 00:56:32,339 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/703ddd1cbced43b287286c523fcea3fa, entries=150, sequenceid=1153, filesize=11.9 K 2023-04-16 00:56:32,339 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp/C/999304bf7b3e4156b92df1e201a7bffc as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/999304bf7b3e4156b92df1e201a7bffc 2023-04-16 00:56:32,342 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/999304bf7b3e4156b92df1e201a7bffc, entries=150, sequenceid=1153, filesize=11.9 K 2023-04-16 00:56:32,343 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=140.89 KB/144270 for d538367277459b70f4a06235dfaacfec in 870ms, sequenceid=1153, compaction requested=true 2023-04-16 00:56:32,343 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for d538367277459b70f4a06235dfaacfec: 2023-04-16 00:56:32,343 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 00:56:32,343 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-16 00:56:32,343 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-16 00:56:32,343 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 00:56:32,343 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 00:56:32,344 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 114572 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-16 00:56:32,344 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HStore(1912): d538367277459b70f4a06235dfaacfec/A is initiating minor compaction (all files) 2023-04-16 00:56:32,344 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of d538367277459b70f4a06235dfaacfec/A in TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. 2023-04-16 00:56:32,344 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A/67fe03bbff1c41b6adeccd6cee68c9e2, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A/b18b0a2fbb8640388db4ee137f17466e, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A/682a8b9d78f9467d8142c539b349897e] into tmpdir=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp, totalSize=111.9 K 2023-04-16 00:56:32,344 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37818 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-16 00:56:32,345 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HStore(1912): d538367277459b70f4a06235dfaacfec/B is initiating minor compaction (all files) 2023-04-16 00:56:32,345 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HRegion(2259): Starting compaction of d538367277459b70f4a06235dfaacfec/B in TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. 2023-04-16 00:56:32,345 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/2e98666f01c24e69a099e8eee156ee6e, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/007c353b19c74958a6b7f5cc4c51fece, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/703ddd1cbced43b287286c523fcea3fa] into tmpdir=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp, totalSize=36.9 K 2023-04-16 00:56:32,346 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.Compactor(207): Compacting 67fe03bbff1c41b6adeccd6cee68c9e2, keycount=150, bloomtype=ROW, size=26.6 K, encoding=NONE, compression=NONE, seqNum=1098, earliestPutTs=1681606589299 2023-04-16 00:56:32,346 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting 2e98666f01c24e69a099e8eee156ee6e, keycount=150, bloomtype=ROW, size=13.1 K, encoding=NONE, compression=NONE, seqNum=1098, earliestPutTs=1681606589299 2023-04-16 00:56:32,346 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.Compactor(207): Compacting b18b0a2fbb8640388db4ee137f17466e, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=1136, earliestPutTs=1681606589333 2023-04-16 00:56:32,346 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting 007c353b19c74958a6b7f5cc4c51fece, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1136, earliestPutTs=1681606589333 2023-04-16 00:56:32,346 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.Compactor(207): Compacting 682a8b9d78f9467d8142c539b349897e, keycount=400, bloomtype=ROW, size=59.8 K, encoding=NONE, compression=NONE, seqNum=1153, earliestPutTs=1681606591458 2023-04-16 00:56:32,347 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting 703ddd1cbced43b287286c523fcea3fa, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1153, earliestPutTs=1681606591467 2023-04-16 00:56:32,353 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] throttle.PressureAwareThroughputController(145): d538367277459b70f4a06235dfaacfec#B#compaction#674 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-16 00:56:32,358 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] throttle.PressureAwareThroughputController(145): d538367277459b70f4a06235dfaacfec#A#compaction#675 average throughput is 4.93 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-16 00:56:32,373 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp/B/f59a5489aa924fe2a05e495dc0ee4e2c as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/f59a5489aa924fe2a05e495dc0ee4e2c 2023-04-16 00:56:32,377 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in d538367277459b70f4a06235dfaacfec/B of d538367277459b70f4a06235dfaacfec into f59a5489aa924fe2a05e495dc0ee4e2c(size=13.2 K), total size for store is 13.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 00:56:32,377 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for d538367277459b70f4a06235dfaacfec: 2023-04-16 00:56:32,377 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec., storeName=d538367277459b70f4a06235dfaacfec/B, priority=13, startTime=1681606592343; duration=0sec 2023-04-16 00:56:32,377 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 00:56:32,377 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-16 00:56:32,378 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37818 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-16 00:56:32,378 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HStore(1912): d538367277459b70f4a06235dfaacfec/C is initiating minor compaction (all files) 2023-04-16 00:56:32,378 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HRegion(2259): Starting compaction of d538367277459b70f4a06235dfaacfec/C in TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. 2023-04-16 00:56:32,378 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/2bbb9b6739aa4649a9850bd0ce9ac919, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/0cfe7ca6635a4b31a3eabc5368f5aec1, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/999304bf7b3e4156b92df1e201a7bffc] into tmpdir=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp, totalSize=36.9 K 2023-04-16 00:56:32,378 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting 2bbb9b6739aa4649a9850bd0ce9ac919, keycount=150, bloomtype=ROW, size=13.1 K, encoding=NONE, compression=NONE, seqNum=1098, earliestPutTs=1681606589299 2023-04-16 00:56:32,379 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting 0cfe7ca6635a4b31a3eabc5368f5aec1, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1136, earliestPutTs=1681606589333 2023-04-16 00:56:32,379 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting 999304bf7b3e4156b92df1e201a7bffc, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1153, earliestPutTs=1681606591467 2023-04-16 00:56:32,384 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] throttle.PressureAwareThroughputController(145): d538367277459b70f4a06235dfaacfec#C#compaction#676 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-16 00:56:32,390 DEBUG [Thread-4138] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:56:32,390 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:56:32,391 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp/C/c4081d2ea77c47b4aeeeb18f883af435 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/c4081d2ea77c47b4aeeeb18f883af435 2023-04-16 00:56:32,392 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:56:32,393 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:56:32,393 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:56:32,393 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:56:32,393 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:56:32,393 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:56:32,393 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:56:32,394 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:32,394 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:32,395 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:32,395 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:56:32,395 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:56:32,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:56:32,395 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:32,395 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:56:32,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:32,395 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:32,396 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in d538367277459b70f4a06235dfaacfec/C of d538367277459b70f4a06235dfaacfec into c4081d2ea77c47b4aeeeb18f883af435(size=13.2 K), total size for store is 13.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 00:56:32,396 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for d538367277459b70f4a06235dfaacfec: 2023-04-16 00:56:32,396 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec., storeName=d538367277459b70f4a06235dfaacfec/C, priority=13, startTime=1681606592343; duration=0sec 2023-04-16 00:56:32,396 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 00:56:32,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:56:32,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:32,396 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:56:32,397 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:56:32,397 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:56:32,397 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:56:32,397 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:56:32,397 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:56:32,397 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:56:32,397 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:56:32,397 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. 2023-04-16 00:56:32,397 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. started... 2023-04-16 00:56:32,397 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing d538367277459b70f4a06235dfaacfec 3/3 column families, dataSize=140.89 KB heapSize=400.22 KB 2023-04-16 00:56:32,397 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d538367277459b70f4a06235dfaacfec, store=A 2023-04-16 00:56:32,397 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 00:56:32,397 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d538367277459b70f4a06235dfaacfec, store=B 2023-04-16 00:56:32,398 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 00:56:32,398 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d538367277459b70f4a06235dfaacfec, store=C 2023-04-16 00:56:32,398 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 00:56:32,422 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230416f0fcc683c9d6440985e55077fe7bb69d to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230416f0fcc683c9d6440985e55077fe7bb69d 2023-04-16 00:56:32,433 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=1192, memsize=47.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp/A/2f08f67213f7492eac8cc1240f416b9e 2023-04-16 00:56:32,443 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=1192 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp/B/a1676505a0984a60834a0f721fd8c557 2023-04-16 00:56:32,452 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=1192 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp/C/3a886566faae4278a01b679f361f642b 2023-04-16 00:56:32,456 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp/A/2f08f67213f7492eac8cc1240f416b9e as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A/2f08f67213f7492eac8cc1240f416b9e 2023-04-16 00:56:32,459 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A/2f08f67213f7492eac8cc1240f416b9e, entries=150, sequenceid=1192, filesize=25.4 K 2023-04-16 00:56:32,460 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp/B/a1676505a0984a60834a0f721fd8c557 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/a1676505a0984a60834a0f721fd8c557 2023-04-16 00:56:32,463 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/a1676505a0984a60834a0f721fd8c557, entries=150, sequenceid=1192, filesize=11.9 K 2023-04-16 00:56:32,464 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp/C/3a886566faae4278a01b679f361f642b as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/3a886566faae4278a01b679f361f642b 2023-04-16 00:56:32,468 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/3a886566faae4278a01b679f361f642b, entries=150, sequenceid=1192, filesize=11.9 K 2023-04-16 00:56:32,468 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~400.17 KB/409776, currentSize=0 B/0 for d538367277459b70f4a06235dfaacfec in 71ms, sequenceid=1192, compaction requested=false 2023-04-16 00:56:32,469 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for d538367277459b70f4a06235dfaacfec: 2023-04-16 00:56:32,469 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. 2023-04-16 00:56:32,469 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:56:32,469 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:56:32,469 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:32,469 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:56:32,469 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:56:32,470 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:32,470 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:32,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:32,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:32,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:32,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:32,471 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:32,471 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:56:32,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:32,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:32,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:32,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:32,472 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:56:32,473 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6d53f0cf[Count = 0] remaining members to acquire global barrier 2023-04-16 00:56:32,473 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:56:32,473 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:32,474 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:32,474 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:32,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:32,474 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:56:32,474 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:56:32,474 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:56:32,474 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:32,474 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:56:32,477 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:32,477 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:56:32,477 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:56:32,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:32,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:32,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:32,477 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:56:32,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:32,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:32,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:32,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:32,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:32,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:32,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:32,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:56:32,479 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:56:32,479 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:56:32,479 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:56:32,479 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:56:32,479 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:56:32,479 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:56:32,481 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:32,481 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:32,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:32,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:32,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:32,481 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:32,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:32,481 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:56:32,482 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:56:32,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:32,482 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:32,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:56:32,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:32,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:32,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:32,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:32,483 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:32,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:32,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:32,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:32,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:32,485 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:56:32,485 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:56:32,485 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:56:32,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:56:32,485 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:32,485 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:32,486 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:32,486 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:32,486 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:32,486 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:32,486 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:32,486 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:56:32,486 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:56:32,486 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:56:32,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:56:32,486 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:56:32,486 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:56:32,486 DEBUG [Thread-4138] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:56:32,486 DEBUG [Thread-4138] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:56:32,586 DEBUG [Thread-4138] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:56:32,587 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:56:32,588 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:56:32,589 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:56:32,606 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:56:32,606 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:56:32,606 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:56:32,606 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:56:32,606 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:56:32,607 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:32,607 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:32,608 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:56:32,608 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:32,608 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:56:32,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:56:32,609 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:32,609 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:56:32,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:32,609 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:32,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:56:32,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:32,609 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:56:32,610 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:56:32,610 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:56:32,610 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:56:32,610 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:56:32,610 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:56:32,610 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. 2023-04-16 00:56:32,610 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. started... 2023-04-16 00:56:32,610 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:56:32,610 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for d538367277459b70f4a06235dfaacfec: 2023-04-16 00:56:32,610 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. 2023-04-16 00:56:32,610 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:56:32,610 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:56:32,610 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:32,610 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:56:32,611 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:56:32,613 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:32,613 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:32,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:32,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:32,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:32,613 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:32,613 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:56:32,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:32,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:32,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:32,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:32,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:32,615 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:56:32,615 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7704364[Count = 0] remaining members to acquire global barrier 2023-04-16 00:56:32,615 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:56:32,615 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:32,616 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:32,616 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:32,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:32,616 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:56:32,616 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:56:32,616 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:56:32,616 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:32,616 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:56:32,618 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:32,618 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:56:32,618 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:56:32,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:32,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:32,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:32,618 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:56:32,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:32,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:32,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:32,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:32,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:32,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:32,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:32,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:56:32,620 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:56:32,620 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:56:32,620 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:56:32,620 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:56:32,620 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:56:32,620 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:56:32,622 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:32,622 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:32,622 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:56:32,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:32,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:32,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:32,622 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:32,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:32,622 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:56:32,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:56:32,622 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:32,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:32,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:32,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:32,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:32,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:32,624 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:32,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:32,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:32,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:32,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:32,627 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:32,627 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:56:32,627 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:56:32,627 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:56:32,627 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:56:32,627 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:56:32,627 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:32,628 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:32,628 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:56:32,627 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:56:32,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:56:32,628 DEBUG [Thread-4138] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:56:32,628 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:32,628 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:32,628 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:32,628 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:32,628 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:56:32,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:56:32,628 DEBUG [Thread-4138] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:56:32,656 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(9158): Flush requested on d538367277459b70f4a06235dfaacfec 2023-04-16 00:56:32,656 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing d538367277459b70f4a06235dfaacfec 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-16 00:56:32,657 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d538367277459b70f4a06235dfaacfec, store=A 2023-04-16 00:56:32,657 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:56:32,657 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d538367277459b70f4a06235dfaacfec, store=B 2023-04-16 00:56:32,657 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:56:32,657 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d538367277459b70f4a06235dfaacfec, store=C 2023-04-16 00:56:32,657 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:56:32,678 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:32,678 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 343 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44554 deadline: 1681606652674, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:32,680 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:32,680 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 289 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44556 deadline: 1681606652677, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:32,680 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:32,680 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 313 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44562 deadline: 1681606652677, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:32,681 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:32,681 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 270 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44564 deadline: 1681606652678, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:32,681 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:32,681 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 379 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44548 deadline: 1681606652678, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:32,729 DEBUG [Thread-4138] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:56:32,729 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:56:32,731 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:56:32,732 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:56:32,732 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:56:32,732 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:56:32,732 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:56:32,732 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:56:32,732 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:56:32,732 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:32,732 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:32,734 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:56:32,734 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:32,734 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:56:32,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:56:32,734 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:32,734 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:56:32,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:32,734 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:32,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:56:32,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:32,735 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:56:32,735 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:56:32,735 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:56:32,735 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:56:32,735 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:56:32,735 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:56:32,735 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:56:32,735 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:56:32,735 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. 2023-04-16 00:56:32,736 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. started... 2023-04-16 00:56:32,736 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. as already flushing 2023-04-16 00:56:32,736 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. 2023-04-16 00:56:32,736 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:56:32,736 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:56:32,736 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:32,736 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:56:32,736 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:56:32,737 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:32,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:32,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:32,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:32,737 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:32,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:32,738 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:32,738 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:56:32,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:32,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:32,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:32,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:32,739 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:56:32,739 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6b96ae28[Count = 0] remaining members to acquire global barrier 2023-04-16 00:56:32,739 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:56:32,739 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:32,740 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:32,740 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:32,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:32,740 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:56:32,740 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:32,740 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:56:32,740 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:56:32,740 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:56:32,742 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:56:32,742 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:32,742 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:56:32,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:32,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:32,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:32,743 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:56:32,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:32,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:32,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:32,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:32,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:32,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:32,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:32,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:56:32,745 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:56:32,745 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:56:32,745 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:56:32,745 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:56:32,745 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:56:32,745 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:56:32,746 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:32,747 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:32,747 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:32,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:32,747 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:56:32,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:32,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:32,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:32,747 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:56:32,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:56:32,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:32,747 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:32,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:32,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:32,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:32,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:32,749 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:32,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:32,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:32,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:32,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:32,751 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:32,751 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:32,751 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:32,751 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:32,751 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:32,751 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:32,751 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:56:32,751 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:56:32,751 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:56:32,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:56:32,751 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:56:32,751 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:56:32,751 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:56:32,751 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:56:32,751 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:32,751 DEBUG [Thread-4138] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:56:32,751 DEBUG [Thread-4138] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:56:32,751 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:56:32,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:56:32,783 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:32,783 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 345 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44554 deadline: 1681606652781, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:32,784 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:32,784 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 291 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44556 deadline: 1681606652781, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:32,784 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:32,785 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 315 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44562 deadline: 1681606652781, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:32,785 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:32,785 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 272 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44564 deadline: 1681606652782, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:32,785 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:32,785 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 381 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44548 deadline: 1681606652782, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:32,787 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp/A/1d0ac54425864bdfa58b8d40561b8e29 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A/1d0ac54425864bdfa58b8d40561b8e29 2023-04-16 00:56:32,793 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in d538367277459b70f4a06235dfaacfec/A of d538367277459b70f4a06235dfaacfec into 1d0ac54425864bdfa58b8d40561b8e29(size=26.7 K), total size for store is 52.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 00:56:32,793 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for d538367277459b70f4a06235dfaacfec: 2023-04-16 00:56:32,793 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec., storeName=d538367277459b70f4a06235dfaacfec/A, priority=13, startTime=1681606592343; duration=0sec 2023-04-16 00:56:32,793 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 00:56:32,851 DEBUG [Thread-4138] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:56:32,852 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:56:32,854 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:56:32,855 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:56:32,855 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:56:32,855 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:56:32,855 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:56:32,855 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:56:32,855 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:56:32,855 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:32,855 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:32,865 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:32,865 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:56:32,865 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:56:32,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:56:32,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:32,866 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:32,866 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:56:32,866 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:32,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:56:32,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:32,866 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:56:32,866 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:56:32,866 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:56:32,867 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:56:32,867 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:56:32,867 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:56:32,867 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:56:32,867 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:56:32,867 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. 2023-04-16 00:56:32,867 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. started... 2023-04-16 00:56:32,867 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. as already flushing 2023-04-16 00:56:32,867 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. 2023-04-16 00:56:32,867 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:56:32,867 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:56:32,867 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:32,867 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:56:32,867 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:56:32,869 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:32,869 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:32,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:32,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:32,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:32,869 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:32,869 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:56:32,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:32,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:32,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:32,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:32,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:32,872 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:56:32,872 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@424f3905[Count = 0] remaining members to acquire global barrier 2023-04-16 00:56:32,872 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:56:32,872 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:32,873 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:32,873 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:32,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:32,873 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:56:32,873 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:56:32,873 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:56:32,873 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:32,874 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:56:32,874 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:56:32,875 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:56:32,874 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:32,875 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:56:32,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:32,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:32,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:32,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:32,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:32,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:32,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:32,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:32,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:32,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:32,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:56:32,877 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:56:32,877 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:56:32,877 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:56:32,877 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:56:32,877 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:56:32,877 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:56:32,880 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:32,880 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:32,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:32,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:32,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:32,880 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:32,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:32,880 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:56:32,881 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:56:32,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:56:32,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:32,881 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:32,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:32,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:32,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:32,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:32,882 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:32,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:32,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:32,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:32,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:32,885 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:56:32,885 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:32,885 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:56:32,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:56:32,885 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:56:32,886 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:56:32,885 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:56:32,885 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:56:32,885 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:32,886 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:32,886 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:32,886 DEBUG [Thread-4138] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:56:32,886 DEBUG [Thread-4138] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:56:32,886 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:56:32,886 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:56:32,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:56:32,886 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:32,886 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:32,887 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:32,986 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:32,986 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 347 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44554 deadline: 1681606652984, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:32,986 DEBUG [Thread-4138] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:56:32,986 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:32,986 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 293 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44556 deadline: 1681606652985, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:32,988 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:56:32,988 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:32,989 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 383 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44548 deadline: 1681606652986, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:32,989 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:32,989 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 274 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44564 deadline: 1681606652989, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:32,990 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:56:32,991 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:56:32,991 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:56:32,991 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:56:32,991 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:56:32,991 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:56:32,991 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:56:32,991 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:32,991 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:32,992 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:32,992 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 317 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44562 deadline: 1681606652990, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:32,993 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:56:32,993 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:56:32,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:56:32,993 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:32,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:32,994 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:32,994 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:56:32,994 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:32,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:56:32,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:32,994 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:56:32,995 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:56:32,995 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:56:32,995 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:56:32,995 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:56:32,995 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:56:32,996 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:56:32,996 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:56:32,996 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. 2023-04-16 00:56:32,996 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. started... 2023-04-16 00:56:32,996 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. as already flushing 2023-04-16 00:56:32,996 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. 2023-04-16 00:56:32,996 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:56:32,996 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:56:32,996 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:32,996 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:56:32,996 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:56:32,997 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:32,997 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:32,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:32,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:32,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:32,998 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:32,998 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:56:32,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:32,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:32,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:32,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:32,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:32,999 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:56:32,999 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@30e87c0b[Count = 0] remaining members to acquire global barrier 2023-04-16 00:56:32,999 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:56:32,999 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:33,002 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:33,002 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:33,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:33,002 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:56:33,002 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:56:33,002 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:56:33,002 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:33,002 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:56:33,003 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:33,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:33,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:33,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:33,003 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:56:33,004 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:56:33,004 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:56:33,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:33,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:33,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:33,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:33,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:33,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:33,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:33,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:56:33,006 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:56:33,006 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:56:33,006 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:56:33,006 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:56:33,006 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:56:33,006 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:56:33,008 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:33,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:33,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:33,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:33,008 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:33,008 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:56:33,008 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:33,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:33,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:33,008 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:56:33,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:56:33,009 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:33,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:33,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:33,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:33,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:33,010 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:33,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:33,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:33,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:33,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:33,012 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:33,012 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:33,012 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:33,012 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:33,012 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:33,012 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:56:33,012 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:56:33,012 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:56:33,012 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:33,012 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:56:33,012 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:56:33,012 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:56:33,012 DEBUG [Thread-4138] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:56:33,012 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:56:33,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:56:33,012 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:33,012 DEBUG [Thread-4138] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:56:33,013 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:56:33,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:56:33,072 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304167af9f126d77f482a8369b7f1ec5f90d5 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304167af9f126d77f482a8369b7f1ec5f90d5 2023-04-16 00:56:33,077 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=1205, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp/A/5c5ede44047045a689058706ef154979 2023-04-16 00:56:33,086 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=1205 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp/B/5455b9b925c2464aa6425061dea32d4a 2023-04-16 00:56:33,094 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=1205 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp/C/f146ab84266e4546ad715eac04e8bd38 2023-04-16 00:56:33,098 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp/A/5c5ede44047045a689058706ef154979 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A/5c5ede44047045a689058706ef154979 2023-04-16 00:56:33,101 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A/5c5ede44047045a689058706ef154979, entries=300, sequenceid=1205, filesize=46.1 K 2023-04-16 00:56:33,101 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp/B/5455b9b925c2464aa6425061dea32d4a as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/5455b9b925c2464aa6425061dea32d4a 2023-04-16 00:56:33,105 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/5455b9b925c2464aa6425061dea32d4a, entries=150, sequenceid=1205, filesize=11.9 K 2023-04-16 00:56:33,106 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp/C/f146ab84266e4546ad715eac04e8bd38 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/f146ab84266e4546ad715eac04e8bd38 2023-04-16 00:56:33,109 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/f146ab84266e4546ad715eac04e8bd38, entries=150, sequenceid=1205, filesize=11.9 K 2023-04-16 00:56:33,109 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for d538367277459b70f4a06235dfaacfec in 453ms, sequenceid=1205, compaction requested=true 2023-04-16 00:56:33,109 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for d538367277459b70f4a06235dfaacfec: 2023-04-16 00:56:33,109 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 00:56:33,110 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-16 00:56:33,110 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-16 00:56:33,110 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 00:56:33,110 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 00:56:33,111 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37920 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-16 00:56:33,111 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 100590 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-16 00:56:33,111 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HStore(1912): d538367277459b70f4a06235dfaacfec/B is initiating minor compaction (all files) 2023-04-16 00:56:33,111 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HStore(1912): d538367277459b70f4a06235dfaacfec/A is initiating minor compaction (all files) 2023-04-16 00:56:33,111 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HRegion(2259): Starting compaction of d538367277459b70f4a06235dfaacfec/B in TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. 2023-04-16 00:56:33,111 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of d538367277459b70f4a06235dfaacfec/A in TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. 2023-04-16 00:56:33,111 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/f59a5489aa924fe2a05e495dc0ee4e2c, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/a1676505a0984a60834a0f721fd8c557, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/5455b9b925c2464aa6425061dea32d4a] into tmpdir=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp, totalSize=37.0 K 2023-04-16 00:56:33,111 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A/1d0ac54425864bdfa58b8d40561b8e29, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A/2f08f67213f7492eac8cc1240f416b9e, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A/5c5ede44047045a689058706ef154979] into tmpdir=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp, totalSize=98.2 K 2023-04-16 00:56:33,112 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting f59a5489aa924fe2a05e495dc0ee4e2c, keycount=150, bloomtype=ROW, size=13.2 K, encoding=NONE, compression=NONE, seqNum=1153, earliestPutTs=1681606591467 2023-04-16 00:56:33,112 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.Compactor(207): Compacting 1d0ac54425864bdfa58b8d40561b8e29, keycount=150, bloomtype=ROW, size=26.7 K, encoding=NONE, compression=NONE, seqNum=1153, earliestPutTs=1681606591467 2023-04-16 00:56:33,112 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting a1676505a0984a60834a0f721fd8c557, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1192, earliestPutTs=1681606591512 2023-04-16 00:56:33,112 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.Compactor(207): Compacting 2f08f67213f7492eac8cc1240f416b9e, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=1192, earliestPutTs=1681606591512 2023-04-16 00:56:33,112 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting 5455b9b925c2464aa6425061dea32d4a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1205, earliestPutTs=1681606592655 2023-04-16 00:56:33,113 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.Compactor(207): Compacting 5c5ede44047045a689058706ef154979, keycount=300, bloomtype=ROW, size=46.1 K, encoding=NONE, compression=NONE, seqNum=1205, earliestPutTs=1681606592649 2023-04-16 00:56:33,113 DEBUG [Thread-4138] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:56:33,113 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:56:33,115 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:56:33,116 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:56:33,116 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:56:33,116 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:56:33,117 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:56:33,117 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:56:33,117 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:56:33,117 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:33,117 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:33,119 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:56:33,119 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:33,119 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:56:33,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:56:33,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:33,120 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:33,120 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:56:33,120 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:33,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:56:33,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:33,121 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:56:33,121 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:56:33,121 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:56:33,121 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:56:33,121 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:56:33,121 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:56:33,121 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:56:33,121 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:56:33,121 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. 2023-04-16 00:56:33,122 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. started... 2023-04-16 00:56:33,122 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing d538367277459b70f4a06235dfaacfec 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-16 00:56:33,122 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d538367277459b70f4a06235dfaacfec, store=A 2023-04-16 00:56:33,122 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:56:33,122 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d538367277459b70f4a06235dfaacfec, store=B 2023-04-16 00:56:33,122 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:56:33,122 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d538367277459b70f4a06235dfaacfec, store=C 2023-04-16 00:56:33,122 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:56:33,127 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] throttle.PressureAwareThroughputController(145): d538367277459b70f4a06235dfaacfec#B#compaction#683 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-16 00:56:33,134 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] throttle.PressureAwareThroughputController(145): d538367277459b70f4a06235dfaacfec#A#compaction#684 average throughput is 2.46 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-16 00:56:33,152 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp/B/78e3e91f3b5e4ea997d8229031a4ed0c as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/78e3e91f3b5e4ea997d8229031a4ed0c 2023-04-16 00:56:33,157 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in d538367277459b70f4a06235dfaacfec/B of d538367277459b70f4a06235dfaacfec into 78e3e91f3b5e4ea997d8229031a4ed0c(size=13.3 K), total size for store is 13.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 00:56:33,157 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for d538367277459b70f4a06235dfaacfec: 2023-04-16 00:56:33,157 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec., storeName=d538367277459b70f4a06235dfaacfec/B, priority=13, startTime=1681606593110; duration=0sec 2023-04-16 00:56:33,158 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 00:56:33,158 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-16 00:56:33,159 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37920 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-16 00:56:33,159 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HStore(1912): d538367277459b70f4a06235dfaacfec/C is initiating minor compaction (all files) 2023-04-16 00:56:33,159 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HRegion(2259): Starting compaction of d538367277459b70f4a06235dfaacfec/C in TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. 2023-04-16 00:56:33,159 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/c4081d2ea77c47b4aeeeb18f883af435, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/3a886566faae4278a01b679f361f642b, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/f146ab84266e4546ad715eac04e8bd38] into tmpdir=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp, totalSize=37.0 K 2023-04-16 00:56:33,160 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting c4081d2ea77c47b4aeeeb18f883af435, keycount=150, bloomtype=ROW, size=13.2 K, encoding=NONE, compression=NONE, seqNum=1153, earliestPutTs=1681606591467 2023-04-16 00:56:33,160 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting 3a886566faae4278a01b679f361f642b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1192, earliestPutTs=1681606591512 2023-04-16 00:56:33,160 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting f146ab84266e4546ad715eac04e8bd38, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1205, earliestPutTs=1681606592655 2023-04-16 00:56:33,161 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230416c45656fe152348eebea749bfa79b4141 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230416c45656fe152348eebea749bfa79b4141 2023-04-16 00:56:33,169 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] throttle.PressureAwareThroughputController(145): d538367277459b70f4a06235dfaacfec#C#compaction#686 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-16 00:56:33,180 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=1226, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp/A/70ce4b2231554814bb0fdde6d896520c 2023-04-16 00:56:33,197 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp/C/d20d89a64451429d9773f8961222a74e as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/d20d89a64451429d9773f8961222a74e 2023-04-16 00:56:33,197 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=1226 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp/B/a17dd28a2c0243fbba5a7c589e5dce26 2023-04-16 00:56:33,202 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in d538367277459b70f4a06235dfaacfec/C of d538367277459b70f4a06235dfaacfec into d20d89a64451429d9773f8961222a74e(size=13.3 K), total size for store is 13.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 00:56:33,202 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for d538367277459b70f4a06235dfaacfec: 2023-04-16 00:56:33,203 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec., storeName=d538367277459b70f4a06235dfaacfec/C, priority=13, startTime=1681606593110; duration=0sec 2023-04-16 00:56:33,203 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 00:56:33,207 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=1226 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp/C/e3a71fdf0aa0416a976512e802b0ad27 2023-04-16 00:56:33,210 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp/A/70ce4b2231554814bb0fdde6d896520c as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A/70ce4b2231554814bb0fdde6d896520c 2023-04-16 00:56:33,214 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A/70ce4b2231554814bb0fdde6d896520c, entries=150, sequenceid=1226, filesize=25.4 K 2023-04-16 00:56:33,214 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp/B/a17dd28a2c0243fbba5a7c589e5dce26 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/a17dd28a2c0243fbba5a7c589e5dce26 2023-04-16 00:56:33,217 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/a17dd28a2c0243fbba5a7c589e5dce26, entries=150, sequenceid=1226, filesize=11.9 K 2023-04-16 00:56:33,218 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp/C/e3a71fdf0aa0416a976512e802b0ad27 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/e3a71fdf0aa0416a976512e802b0ad27 2023-04-16 00:56:33,221 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/e3a71fdf0aa0416a976512e802b0ad27, entries=150, sequenceid=1226, filesize=11.9 K 2023-04-16 00:56:33,222 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=0 B/0 for d538367277459b70f4a06235dfaacfec in 100ms, sequenceid=1226, compaction requested=false 2023-04-16 00:56:33,222 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for d538367277459b70f4a06235dfaacfec: 2023-04-16 00:56:33,222 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. 2023-04-16 00:56:33,222 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:56:33,222 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:56:33,222 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:33,222 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:56:33,223 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:56:33,224 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:33,224 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:33,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:33,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:33,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:33,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:33,224 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:33,225 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:56:33,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:33,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:33,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:33,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:33,226 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:56:33,226 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@35866211[Count = 0] remaining members to acquire global barrier 2023-04-16 00:56:33,226 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:56:33,226 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:33,227 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:33,227 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:33,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:33,227 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:56:33,227 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:56:33,227 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:56:33,227 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:33,227 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:56:33,229 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:33,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:33,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:33,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:33,229 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:56:33,229 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:56:33,229 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:56:33,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:33,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:33,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:33,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:33,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:33,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:33,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:33,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:56:33,231 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:56:33,231 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:56:33,231 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:56:33,231 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:56:33,231 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:56:33,231 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:56:33,234 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:33,234 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:33,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:33,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:33,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:33,234 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:33,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:33,234 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:56:33,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:33,235 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:56:33,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:56:33,235 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:33,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:33,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:33,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:33,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:33,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:33,237 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:33,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:33,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:33,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:33,239 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:33,239 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:56:33,239 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:56:33,239 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:33,239 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:56:33,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:56:33,239 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:33,239 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:56:33,239 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:33,239 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:56:33,240 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:56:33,240 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:56:33,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:56:33,240 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:33,240 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:33,240 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:33,240 DEBUG [Thread-4138] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:56:33,240 DEBUG [Thread-4138] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:56:33,311 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(9158): Flush requested on d538367277459b70f4a06235dfaacfec 2023-04-16 00:56:33,311 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing d538367277459b70f4a06235dfaacfec 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-16 00:56:33,311 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d538367277459b70f4a06235dfaacfec, store=A 2023-04-16 00:56:33,312 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:56:33,312 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d538367277459b70f4a06235dfaacfec, store=B 2023-04-16 00:56:33,312 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:56:33,312 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d538367277459b70f4a06235dfaacfec, store=C 2023-04-16 00:56:33,312 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:56:33,340 DEBUG [Thread-4138] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:56:33,341 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:56:33,342 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:56:33,343 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:56:33,343 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:56:33,343 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:56:33,343 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:56:33,343 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:56:33,343 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:56:33,343 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:33,343 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:33,344 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:33,344 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:56:33,344 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:56:33,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:56:33,345 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:33,345 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:56:33,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:33,345 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:33,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:56:33,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:33,345 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:56:33,346 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:56:33,346 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:56:33,346 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:56:33,346 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:56:33,346 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:56:33,346 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:56:33,346 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:56:33,346 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. 2023-04-16 00:56:33,346 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. started... 2023-04-16 00:56:33,346 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. as already flushing 2023-04-16 00:56:33,347 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. 2023-04-16 00:56:33,347 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:56:33,347 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:56:33,347 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:33,347 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:56:33,347 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:56:33,351 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:33,351 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:33,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:33,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:33,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:33,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:33,352 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:33,352 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:56:33,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:33,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:33,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:33,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:33,354 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:56:33,354 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6b3f7d67[Count = 0] remaining members to acquire global barrier 2023-04-16 00:56:33,355 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14144, totalHeapSize=32376, min timestamp=1681606593321, max timestamp=1681606593353Num uniques 859; 2023-04-16 00:56:33,355 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=14144, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-16 00:56:33,355 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14144, totalHeapSize=32376, min timestamp=1681606593321, max timestamp=1681606593353Num uniques 859; 2023-04-16 00:56:33,355 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=14144, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-16 00:56:33,354 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:56:33,356 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14144, totalHeapSize=32376, min timestamp=1681606593321, max timestamp=1681606593353Num uniques 859; 2023-04-16 00:56:33,356 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=14144, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-16 00:56:33,356 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:33,357 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:33,357 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:33,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:33,357 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:56:33,357 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:56:33,357 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:56:33,358 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:33,358 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:56:33,359 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:56:33,359 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:56:33,359 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:33,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:33,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:33,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:33,359 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:56:33,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:33,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:33,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:33,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:33,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:33,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:33,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:33,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:56:33,363 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:56:33,363 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:56:33,363 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:56:33,363 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:56:33,363 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:56:33,363 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:56:33,364 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:33,364 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:33,364 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:33,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:33,364 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:56:33,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:33,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:33,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:33,365 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:56:33,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:56:33,365 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:33,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:33,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:33,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:33,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:33,366 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:33,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:33,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:33,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:33,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:33,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:33,370 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:56:33,370 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:56:33,370 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:33,370 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:56:33,370 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:56:33,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:56:33,370 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:33,370 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:56:33,370 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:33,370 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:56:33,370 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:56:33,370 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:33,371 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:33,371 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:33,371 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:33,371 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:56:33,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:56:33,371 DEBUG [Thread-4138] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:56:33,371 DEBUG [Thread-4138] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:56:33,374 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:33,374 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 303 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44556 deadline: 1681606653369, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:33,375 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:33,375 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 393 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44548 deadline: 1681606653372, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:33,375 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:33,376 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 357 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44554 deadline: 1681606653373, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:33,378 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:33,378 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 284 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44564 deadline: 1681606653375, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:33,379 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:33,379 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 327 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44562 deadline: 1681606653375, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:33,471 DEBUG [Thread-4138] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:56:33,471 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:56:33,473 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:56:33,474 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:56:33,474 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:56:33,474 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:56:33,474 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:56:33,474 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:56:33,474 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:56:33,475 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:33,475 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:33,476 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:33,476 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:56:33,476 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:56:33,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:56:33,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:33,478 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:33,478 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 305 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44556 deadline: 1681606653475, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:33,478 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:33,478 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:56:33,478 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:33,478 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:33,478 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 395 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44548 deadline: 1681606653476, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:33,478 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:33,478 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 359 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44554 deadline: 1681606653477, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:33,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:56:33,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:33,479 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:56:33,479 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:56:33,479 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:56:33,479 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:56:33,479 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:56:33,479 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:56:33,479 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:56:33,479 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:56:33,479 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. 2023-04-16 00:56:33,479 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. started... 2023-04-16 00:56:33,479 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. as already flushing 2023-04-16 00:56:33,479 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. 2023-04-16 00:56:33,480 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:56:33,480 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:56:33,480 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:33,480 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:56:33,480 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:56:33,480 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:33,480 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 286 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44564 deadline: 1681606653479, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:33,481 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:33,481 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:33,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:33,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:33,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:33,482 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:33,482 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:56:33,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:33,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:33,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:33,483 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:33,483 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 329 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44562 deadline: 1681606653481, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:33,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:33,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:33,484 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:56:33,484 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5003f614[Count = 0] remaining members to acquire global barrier 2023-04-16 00:56:33,484 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:56:33,484 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:33,485 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:33,485 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:33,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:33,485 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:56:33,485 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:56:33,485 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:33,485 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:56:33,485 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:56:33,488 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:33,488 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:56:33,488 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:56:33,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:33,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:33,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:33,488 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:56:33,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:33,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:33,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:33,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:33,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:33,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:33,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:33,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:56:33,490 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:56:33,490 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:56:33,491 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:56:33,491 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:56:33,491 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:56:33,491 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:56:33,492 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:33,492 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:56:33,492 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:33,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:33,492 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:33,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:33,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:33,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:33,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:33,493 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:33,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:33,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:33,494 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:56:33,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:56:33,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:33,494 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:33,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:33,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:33,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:33,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:33,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:33,498 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:56:33,498 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:56:33,498 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:33,498 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:33,498 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:33,498 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:33,498 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:33,498 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:33,498 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:33,498 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:56:33,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:56:33,498 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:56:33,498 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:56:33,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:56:33,498 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:56:33,498 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:56:33,499 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:56:33,499 DEBUG [Thread-4138] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:56:33,499 DEBUG [Thread-4138] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:56:33,577 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp/A/175324f03802417cb8a838e0046c05a4 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A/175324f03802417cb8a838e0046c05a4 2023-04-16 00:56:33,581 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in d538367277459b70f4a06235dfaacfec/A of d538367277459b70f4a06235dfaacfec into 175324f03802417cb8a838e0046c05a4(size=26.8 K), total size for store is 52.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 00:56:33,582 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for d538367277459b70f4a06235dfaacfec: 2023-04-16 00:56:33,582 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec., storeName=d538367277459b70f4a06235dfaacfec/A, priority=13, startTime=1681606593109; duration=0sec 2023-04-16 00:56:33,582 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 00:56:33,599 DEBUG [Thread-4138] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:56:33,599 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:56:33,601 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:56:33,602 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:56:33,602 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:56:33,602 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:56:33,602 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:56:33,602 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:56:33,602 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:56:33,602 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:33,602 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:33,603 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:56:33,604 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:33,604 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:56:33,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:56:33,604 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:33,604 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:56:33,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:33,604 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:33,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:56:33,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:33,605 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:56:33,605 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:56:33,605 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:56:33,605 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:56:33,605 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:56:33,605 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:56:33,605 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:56:33,605 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. 2023-04-16 00:56:33,605 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. started... 2023-04-16 00:56:33,605 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:56:33,605 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. as already flushing 2023-04-16 00:56:33,606 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. 2023-04-16 00:56:33,606 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:56:33,606 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:56:33,606 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:33,606 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:56:33,606 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:56:33,608 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:33,608 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:33,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:33,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:33,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:33,608 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:33,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:33,608 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:56:33,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:33,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:33,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:33,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:33,610 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:56:33,610 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@23f8d03b[Count = 0] remaining members to acquire global barrier 2023-04-16 00:56:33,610 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:56:33,610 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:33,611 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:33,611 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:33,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:33,611 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:56:33,611 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:56:33,611 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:56:33,611 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:33,611 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:56:33,612 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:33,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:33,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:33,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:33,612 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:56:33,613 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:56:33,613 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:56:33,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:33,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:33,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:33,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:33,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:33,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:33,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:33,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:56:33,615 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:56:33,615 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:56:33,615 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:56:33,615 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:56:33,615 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:56:33,615 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:56:33,616 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:33,617 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:56:33,617 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:33,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:33,617 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:33,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:33,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:33,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:33,617 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:56:33,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:56:33,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:33,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:33,618 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:33,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:33,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:33,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:33,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:33,619 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:33,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:33,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:33,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:33,622 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:56:33,622 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:56:33,622 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:56:33,622 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:56:33,622 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:56:33,622 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:56:33,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:56:33,622 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:56:33,622 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:33,622 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:33,622 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:33,622 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:33,622 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:33,622 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:33,622 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:33,622 DEBUG [Thread-4138] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:56:33,622 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:56:33,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:56:33,622 DEBUG [Thread-4138] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:56:33,681 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:33,681 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 307 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44556 deadline: 1681606653679, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:33,681 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:33,681 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 397 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44548 deadline: 1681606653679, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:33,682 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:33,682 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 361 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44554 deadline: 1681606653679, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:33,684 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:33,684 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 288 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44564 deadline: 1681606653681, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:33,688 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:33,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 331 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44562 deadline: 1681606653685, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:33,723 DEBUG [Thread-4138] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:56:33,723 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:56:33,724 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:56:33,726 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:56:33,726 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:56:33,726 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:56:33,726 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:56:33,726 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:56:33,726 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:56:33,726 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:33,726 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:33,727 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:56:33,727 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:56:33,727 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:33,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:56:33,728 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:33,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:33,728 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:56:33,728 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:33,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:56:33,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:33,729 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:56:33,729 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:56:33,729 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:56:33,729 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:56:33,729 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:56:33,729 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:56:33,729 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:56:33,729 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. 2023-04-16 00:56:33,730 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. started... 2023-04-16 00:56:33,729 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:56:33,730 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. as already flushing 2023-04-16 00:56:33,730 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304163a567c6cbeea40d398b6ee1b5c52932a to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304163a567c6cbeea40d398b6ee1b5c52932a 2023-04-16 00:56:33,730 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. 2023-04-16 00:56:33,730 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:56:33,730 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:56:33,730 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:33,730 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:56:33,730 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:56:33,732 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:33,732 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:33,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:33,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:33,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:33,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:33,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:33,732 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:33,732 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:56:33,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:33,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:33,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:33,733 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:56:33,733 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@25e8b8bf[Count = 0] remaining members to acquire global barrier 2023-04-16 00:56:33,733 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:56:33,733 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:33,735 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:33,735 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:33,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:33,735 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:56:33,735 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:56:33,735 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:56:33,735 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=1238, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp/A/f16aacb5158b4551b6b5e6903ff792b3 2023-04-16 00:56:33,735 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:33,735 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:56:33,741 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:33,741 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:56:33,741 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:56:33,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:33,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:33,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:33,741 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:56:33,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:33,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:33,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:33,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:33,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:33,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:33,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:33,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:56:33,744 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:56:33,744 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:56:33,744 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:56:33,744 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:56:33,744 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:56:33,744 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:56:33,745 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=1238 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp/B/fb0afb1309664563a14a31e108cd473b 2023-04-16 00:56:33,747 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:33,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:33,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:33,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:33,747 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:33,747 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:56:33,747 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:33,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:33,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:33,747 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:56:33,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:56:33,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:33,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:33,748 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:33,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:33,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:33,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:33,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:33,750 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:33,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:33,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:33,755 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:56:33,755 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:56:33,755 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:56:33,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:56:33,755 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:56:33,755 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:56:33,755 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:33,755 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:33,755 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:56:33,755 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:33,755 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:33,755 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:33,755 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:56:33,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:56:33,755 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:33,755 DEBUG [Thread-4138] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:56:33,755 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:33,755 DEBUG [Thread-4138] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:56:33,761 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=1238 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp/C/6b64c1550c544cd099f3c8a1b84b780d 2023-04-16 00:56:33,765 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp/A/f16aacb5158b4551b6b5e6903ff792b3 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A/f16aacb5158b4551b6b5e6903ff792b3 2023-04-16 00:56:33,769 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A/f16aacb5158b4551b6b5e6903ff792b3, entries=350, sequenceid=1238, filesize=52.9 K 2023-04-16 00:56:33,769 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp/B/fb0afb1309664563a14a31e108cd473b as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/fb0afb1309664563a14a31e108cd473b 2023-04-16 00:56:33,772 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/fb0afb1309664563a14a31e108cd473b, entries=150, sequenceid=1238, filesize=11.9 K 2023-04-16 00:56:33,773 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp/C/6b64c1550c544cd099f3c8a1b84b780d as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/6b64c1550c544cd099f3c8a1b84b780d 2023-04-16 00:56:33,776 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/6b64c1550c544cd099f3c8a1b84b780d, entries=150, sequenceid=1238, filesize=11.9 K 2023-04-16 00:56:33,777 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=147.60 KB/151140 for d538367277459b70f4a06235dfaacfec in 466ms, sequenceid=1238, compaction requested=true 2023-04-16 00:56:33,777 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for d538367277459b70f4a06235dfaacfec: 2023-04-16 00:56:33,777 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 00:56:33,777 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-16 00:56:33,777 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-16 00:56:33,777 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 00:56:33,778 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 00:56:33,778 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 107736 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-16 00:56:33,778 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 38022 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-16 00:56:33,778 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HStore(1912): d538367277459b70f4a06235dfaacfec/A is initiating minor compaction (all files) 2023-04-16 00:56:33,778 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HStore(1912): d538367277459b70f4a06235dfaacfec/B is initiating minor compaction (all files) 2023-04-16 00:56:33,779 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of d538367277459b70f4a06235dfaacfec/A in TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. 2023-04-16 00:56:33,779 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HRegion(2259): Starting compaction of d538367277459b70f4a06235dfaacfec/B in TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. 2023-04-16 00:56:33,779 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A/175324f03802417cb8a838e0046c05a4, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A/70ce4b2231554814bb0fdde6d896520c, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A/f16aacb5158b4551b6b5e6903ff792b3] into tmpdir=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp, totalSize=105.2 K 2023-04-16 00:56:33,779 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/78e3e91f3b5e4ea997d8229031a4ed0c, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/a17dd28a2c0243fbba5a7c589e5dce26, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/fb0afb1309664563a14a31e108cd473b] into tmpdir=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp, totalSize=37.1 K 2023-04-16 00:56:33,779 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.Compactor(207): Compacting 175324f03802417cb8a838e0046c05a4, keycount=150, bloomtype=ROW, size=26.8 K, encoding=NONE, compression=NONE, seqNum=1205, earliestPutTs=1681606592655 2023-04-16 00:56:33,779 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting 78e3e91f3b5e4ea997d8229031a4ed0c, keycount=150, bloomtype=ROW, size=13.3 K, encoding=NONE, compression=NONE, seqNum=1205, earliestPutTs=1681606592655 2023-04-16 00:56:33,779 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.Compactor(207): Compacting 70ce4b2231554814bb0fdde6d896520c, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=1226, earliestPutTs=1681606592671 2023-04-16 00:56:33,779 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting a17dd28a2c0243fbba5a7c589e5dce26, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1226, earliestPutTs=1681606592671 2023-04-16 00:56:33,780 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.Compactor(207): Compacting f16aacb5158b4551b6b5e6903ff792b3, keycount=350, bloomtype=ROW, size=52.9 K, encoding=NONE, compression=NONE, seqNum=1238, earliestPutTs=1681606593302 2023-04-16 00:56:33,780 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting fb0afb1309664563a14a31e108cd473b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1238, earliestPutTs=1681606593302 2023-04-16 00:56:33,789 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] throttle.PressureAwareThroughputController(145): d538367277459b70f4a06235dfaacfec#B#compaction#693 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-16 00:56:33,803 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] throttle.PressureAwareThroughputController(145): d538367277459b70f4a06235dfaacfec#A#compaction#692 average throughput is 1.31 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-16 00:56:33,818 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp/B/4a9ae918db844b738da120a5a9eef156 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/4a9ae918db844b738da120a5a9eef156 2023-04-16 00:56:33,823 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in d538367277459b70f4a06235dfaacfec/B of d538367277459b70f4a06235dfaacfec into 4a9ae918db844b738da120a5a9eef156(size=13.4 K), total size for store is 13.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 00:56:33,823 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for d538367277459b70f4a06235dfaacfec: 2023-04-16 00:56:33,823 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec., storeName=d538367277459b70f4a06235dfaacfec/B, priority=13, startTime=1681606593777; duration=0sec 2023-04-16 00:56:33,824 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 00:56:33,824 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-16 00:56:33,825 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 38022 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-16 00:56:33,825 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HStore(1912): d538367277459b70f4a06235dfaacfec/C is initiating minor compaction (all files) 2023-04-16 00:56:33,825 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HRegion(2259): Starting compaction of d538367277459b70f4a06235dfaacfec/C in TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. 2023-04-16 00:56:33,825 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/d20d89a64451429d9773f8961222a74e, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/e3a71fdf0aa0416a976512e802b0ad27, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/6b64c1550c544cd099f3c8a1b84b780d] into tmpdir=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp, totalSize=37.1 K 2023-04-16 00:56:33,826 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting d20d89a64451429d9773f8961222a74e, keycount=150, bloomtype=ROW, size=13.3 K, encoding=NONE, compression=NONE, seqNum=1205, earliestPutTs=1681606592655 2023-04-16 00:56:33,827 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting e3a71fdf0aa0416a976512e802b0ad27, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1226, earliestPutTs=1681606592671 2023-04-16 00:56:33,828 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting 6b64c1550c544cd099f3c8a1b84b780d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1238, earliestPutTs=1681606593302 2023-04-16 00:56:33,840 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] throttle.PressureAwareThroughputController(145): d538367277459b70f4a06235dfaacfec#C#compaction#694 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-16 00:56:33,847 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp/A/91feb7f67cc64dabbd8922d9cf8d3c5a as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A/91feb7f67cc64dabbd8922d9cf8d3c5a 2023-04-16 00:56:33,852 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in d538367277459b70f4a06235dfaacfec/A of d538367277459b70f4a06235dfaacfec into 91feb7f67cc64dabbd8922d9cf8d3c5a(size=26.9 K), total size for store is 26.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 00:56:33,852 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for d538367277459b70f4a06235dfaacfec: 2023-04-16 00:56:33,852 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec., storeName=d538367277459b70f4a06235dfaacfec/A, priority=13, startTime=1681606593777; duration=0sec 2023-04-16 00:56:33,852 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 00:56:33,856 DEBUG [Thread-4138] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:56:33,856 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:56:33,858 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:56:33,860 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:56:33,860 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:56:33,860 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:56:33,861 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:56:33,861 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:56:33,861 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:56:33,861 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:33,861 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:33,862 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:56:33,862 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:33,862 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:56:33,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:56:33,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:33,863 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:33,863 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:56:33,863 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:33,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:56:33,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:33,864 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:56:33,872 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:56:33,872 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:56:33,872 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:56:33,872 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:56:33,872 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:56:33,878 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp/C/4492b0b9e8c64fef9380ae8a5f8af5ad as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/4492b0b9e8c64fef9380ae8a5f8af5ad 2023-04-16 00:56:33,880 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:56:33,880 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:56:33,880 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. 2023-04-16 00:56:33,880 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. started... 2023-04-16 00:56:33,880 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing d538367277459b70f4a06235dfaacfec 3/3 column families, dataSize=147.60 KB heapSize=410.77 KB 2023-04-16 00:56:33,880 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d538367277459b70f4a06235dfaacfec, store=A 2023-04-16 00:56:33,880 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 00:56:33,881 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d538367277459b70f4a06235dfaacfec, store=B 2023-04-16 00:56:33,881 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 00:56:33,881 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d538367277459b70f4a06235dfaacfec, store=C 2023-04-16 00:56:33,881 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 00:56:33,883 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in d538367277459b70f4a06235dfaacfec/C of d538367277459b70f4a06235dfaacfec into 4492b0b9e8c64fef9380ae8a5f8af5ad(size=13.4 K), total size for store is 13.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 00:56:33,883 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for d538367277459b70f4a06235dfaacfec: 2023-04-16 00:56:33,883 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec., storeName=d538367277459b70f4a06235dfaacfec/C, priority=13, startTime=1681606593778; duration=0sec 2023-04-16 00:56:33,883 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 00:56:33,898 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230416e6aa741db86d40419795df5682b3ffff to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230416e6aa741db86d40419795df5682b3ffff 2023-04-16 00:56:33,905 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=1277, memsize=49.2 K, hasBloomFilter=true, into tmp file hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp/A/3532e4998ae246be82011ecdb3cb9f94 2023-04-16 00:56:33,919 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=1277 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp/B/f8a86b4f975d46ce9c525645e6a5aeb5 2023-04-16 00:56:33,941 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=1277 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp/C/ea3f2d4b75654f28aeca92dc38cec363 2023-04-16 00:56:33,945 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp/A/3532e4998ae246be82011ecdb3cb9f94 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A/3532e4998ae246be82011ecdb3cb9f94 2023-04-16 00:56:33,948 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A/3532e4998ae246be82011ecdb3cb9f94, entries=150, sequenceid=1277, filesize=25.4 K 2023-04-16 00:56:33,949 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp/B/f8a86b4f975d46ce9c525645e6a5aeb5 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/f8a86b4f975d46ce9c525645e6a5aeb5 2023-04-16 00:56:33,953 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/f8a86b4f975d46ce9c525645e6a5aeb5, entries=150, sequenceid=1277, filesize=11.9 K 2023-04-16 00:56:33,954 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp/C/ea3f2d4b75654f28aeca92dc38cec363 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/ea3f2d4b75654f28aeca92dc38cec363 2023-04-16 00:56:33,958 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/ea3f2d4b75654f28aeca92dc38cec363, entries=150, sequenceid=1277, filesize=11.9 K 2023-04-16 00:56:33,958 INFO [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~147.60 KB/151140, heapSize ~410.72 KB/420576, currentSize=0 B/0 for d538367277459b70f4a06235dfaacfec in 78ms, sequenceid=1277, compaction requested=false 2023-04-16 00:56:33,958 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for d538367277459b70f4a06235dfaacfec: 2023-04-16 00:56:33,959 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. 2023-04-16 00:56:33,959 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:56:33,959 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:56:33,959 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:33,959 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:56:33,959 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:56:33,962 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:33,962 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:33,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:33,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:33,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:33,962 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:33,962 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:56:33,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:33,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:33,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:33,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:33,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:33,964 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:56:33,964 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@352c2d20[Count = 0] remaining members to acquire global barrier 2023-04-16 00:56:33,964 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:56:33,964 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:33,965 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:33,965 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:33,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:33,965 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:56:33,965 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:56:33,965 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:33,965 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:56:33,965 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:56:33,967 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:33,967 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:56:33,967 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:56:33,967 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:56:33,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:33,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:33,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:33,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:33,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:33,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:33,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:33,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:33,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:33,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:33,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:56:33,970 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:56:33,970 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:56:33,970 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:56:33,970 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:56:33,970 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:56:33,970 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:56:33,971 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:33,971 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:33,971 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:56:33,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:33,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:33,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:33,971 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:33,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:33,972 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:56:33,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:56:33,972 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:33,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:33,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:33,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:33,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:33,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:33,973 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:33,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:33,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:33,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:33,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:33,977 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:33,977 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:33,977 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:33,977 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:33,977 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:33,977 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:56:33,977 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:56:33,977 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:56:33,977 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:56:33,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:56:33,977 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:56:33,977 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:56:33,977 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:56:33,978 DEBUG [Thread-4138] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:56:33,978 DEBUG [Thread-4138] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:56:33,977 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:33,978 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:33,978 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:56:33,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:56:34,018 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(9158): Flush requested on d538367277459b70f4a06235dfaacfec 2023-04-16 00:56:34,018 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing d538367277459b70f4a06235dfaacfec 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-16 00:56:34,019 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d538367277459b70f4a06235dfaacfec, store=A 2023-04-16 00:56:34,019 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:56:34,019 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d538367277459b70f4a06235dfaacfec, store=B 2023-04-16 00:56:34,019 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:56:34,019 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d538367277459b70f4a06235dfaacfec, store=C 2023-04-16 00:56:34,019 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:56:34,043 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:34,043 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 404 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44548 deadline: 1681606654040, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:34,044 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681606594035, max timestamp=1681606594042Num uniques 859; 2023-04-16 00:56:34,044 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-16 00:56:34,044 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681606594035, max timestamp=1681606594042Num uniques 859; 2023-04-16 00:56:34,044 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-16 00:56:34,044 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681606594035, max timestamp=1681606594042Num uniques 859; 2023-04-16 00:56:34,044 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-16 00:56:34,059 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:34,059 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 341 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44562 deadline: 1681606654056, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:34,059 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:34,059 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 372 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44554 deadline: 1681606654057, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:34,061 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:34,062 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 318 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44556 deadline: 1681606654058, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:34,062 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:34,062 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 299 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44564 deadline: 1681606654059, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:34,078 DEBUG [Thread-4138] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:56:34,078 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:56:34,079 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:56:34,080 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:56:34,081 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:56:34,081 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:56:34,081 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:56:34,081 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:56:34,081 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:56:34,081 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:34,081 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:34,082 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:56:34,082 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:34,082 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:56:34,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:56:34,083 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:34,083 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:56:34,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:34,083 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:34,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:56:34,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:34,083 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:56:34,084 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:56:34,084 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:56:34,084 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:56:34,084 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:56:34,084 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:56:34,084 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:56:34,084 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:56:34,084 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. 2023-04-16 00:56:34,084 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. started... 2023-04-16 00:56:34,085 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. as already flushing 2023-04-16 00:56:34,085 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. 2023-04-16 00:56:34,085 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:56:34,085 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:56:34,085 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:34,085 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:56:34,085 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:56:34,086 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:34,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:34,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:34,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:34,086 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:34,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:34,087 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:34,087 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:56:34,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:34,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:34,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:34,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:34,088 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:56:34,088 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7ad0cec7[Count = 0] remaining members to acquire global barrier 2023-04-16 00:56:34,088 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:56:34,088 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:34,089 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:34,089 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:34,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:34,089 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:56:34,089 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:56:34,089 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:56:34,089 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:34,089 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:56:34,091 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:56:34,091 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:34,091 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:56:34,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:34,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:34,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:34,091 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:56:34,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:34,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:34,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:34,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:34,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:34,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:34,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:34,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:56:34,093 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:56:34,093 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:56:34,093 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:56:34,093 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:56:34,093 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:56:34,093 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:56:34,095 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:34,095 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:34,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:34,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:34,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:34,095 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:34,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:34,095 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:56:34,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:34,096 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:56:34,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:56:34,096 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:34,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:34,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:34,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:34,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:34,097 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:34,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:34,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:34,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:34,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:34,099 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:34,099 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:34,099 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:34,099 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:34,100 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:56:34,100 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:56:34,100 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:56:34,100 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:56:34,100 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:56:34,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:56:34,100 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:56:34,100 DEBUG [Thread-4138] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:56:34,100 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:34,100 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:56:34,100 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:34,100 DEBUG [Thread-4138] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:56:34,100 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:34,100 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:56:34,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:56:34,146 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:34,146 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 406 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44548 deadline: 1681606654145, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:34,162 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:34,162 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 343 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44562 deadline: 1681606654160, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:34,162 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:34,162 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 374 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44554 deadline: 1681606654160, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:34,163 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:34,164 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 301 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44564 deadline: 1681606654163, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:34,164 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:34,164 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 320 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44556 deadline: 1681606654163, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:34,200 DEBUG [Thread-4138] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:56:34,201 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:56:34,202 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:56:34,204 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:56:34,204 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:56:34,204 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:56:34,204 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:56:34,204 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:56:34,204 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:56:34,204 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:34,204 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:34,206 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:56:34,206 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:34,206 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:56:34,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:56:34,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:34,206 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:34,206 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:56:34,207 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:34,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:56:34,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:34,207 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:56:34,207 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:56:34,207 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:56:34,207 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:56:34,208 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:56:34,208 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:56:34,208 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:56:34,208 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:56:34,208 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. 2023-04-16 00:56:34,208 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. started... 2023-04-16 00:56:34,208 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. as already flushing 2023-04-16 00:56:34,208 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. 2023-04-16 00:56:34,208 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:56:34,208 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:56:34,208 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:34,208 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:56:34,208 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:56:34,211 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:34,211 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:34,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:34,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:34,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:34,211 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:34,212 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:56:34,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:34,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:34,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:34,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:34,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:34,213 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:56:34,213 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1fccb0ae[Count = 0] remaining members to acquire global barrier 2023-04-16 00:56:34,213 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:56:34,213 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:34,214 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:34,214 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:34,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:34,214 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:56:34,214 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:56:34,214 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:56:34,215 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:34,215 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:56:34,216 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:34,216 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:56:34,216 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:56:34,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:34,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:34,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:34,216 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:56:34,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:34,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:34,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:34,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:34,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:34,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:34,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:34,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:56:34,219 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:56:34,219 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:56:34,219 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:56:34,219 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:56:34,219 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:56:34,219 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:56:34,224 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:34,224 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:34,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:34,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:34,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:34,224 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:34,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:34,224 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:56:34,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:34,225 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:56:34,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:56:34,225 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:34,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:34,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:34,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:34,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:34,226 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:34,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:34,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:34,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:34,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:34,230 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:56:34,230 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:56:34,230 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:56:34,230 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:34,230 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:34,230 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:34,230 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:56:34,230 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:56:34,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:56:34,230 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:34,230 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:56:34,231 DEBUG [Thread-4138] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:56:34,231 DEBUG [Thread-4138] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:56:34,231 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:34,231 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:56:34,231 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:56:34,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:56:34,231 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:34,231 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:34,331 DEBUG [Thread-4138] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:56:34,331 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:56:34,333 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:56:34,334 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:56:34,334 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:56:34,334 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:56:34,334 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:56:34,334 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:56:34,334 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:56:34,335 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:34,335 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:34,336 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:56:34,336 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:34,336 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:56:34,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:56:34,336 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:34,336 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:56:34,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:34,337 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:34,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:56:34,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:34,337 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:56:34,337 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:56:34,337 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:56:34,337 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:56:34,338 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:56:34,338 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:56:34,338 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. 2023-04-16 00:56:34,338 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. started... 2023-04-16 00:56:34,338 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:56:34,338 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. as already flushing 2023-04-16 00:56:34,338 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. 2023-04-16 00:56:34,338 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:56:34,338 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:56:34,338 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:34,338 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:56:34,338 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:56:34,340 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:34,340 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:34,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:34,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:34,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:34,341 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:34,341 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:56:34,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:34,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:34,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:34,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:34,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:34,343 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:56:34,343 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7f81dea8[Count = 0] remaining members to acquire global barrier 2023-04-16 00:56:34,343 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:56:34,343 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:34,344 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:34,344 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:34,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:34,344 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:56:34,345 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:56:34,345 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:56:34,345 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:34,345 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:56:34,346 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:34,346 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:56:34,346 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:56:34,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:34,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:34,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:34,346 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:56:34,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:34,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:34,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:34,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:34,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:34,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:34,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:34,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:56:34,350 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:56:34,350 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:34,350 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:56:34,350 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:56:34,350 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:56:34,350 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:56:34,350 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:56:34,350 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 408 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44548 deadline: 1681606654348, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:34,352 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:34,352 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:34,352 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:34,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:34,352 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:56:34,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:34,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:34,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:34,352 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:56:34,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:56:34,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:34,352 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:34,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:34,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:34,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:34,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:34,353 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:34,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:34,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:34,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:34,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:34,358 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:56:34,358 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:56:34,358 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:56:34,358 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:34,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:56:34,358 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:56:34,358 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:34,358 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:34,358 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:34,358 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:34,358 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:34,358 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:56:34,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:56:34,358 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:34,360 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:56:34,360 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:56:34,360 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:56:34,360 DEBUG [Thread-4138] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:56:34,360 DEBUG [Thread-4138] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:56:34,363 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:34,363 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 376 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44554 deadline: 1681606654363, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:34,365 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:34,365 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 345 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44562 deadline: 1681606654364, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:34,366 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:34,366 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:34,366 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 303 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44564 deadline: 1681606654365, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:34,366 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34975] ipc.CallRunner(144): callId: 322 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44556 deadline: 1681606654365, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:34,434 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304166b901ddc4c194501844edcf1d3282c3b to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304166b901ddc4c194501844edcf1d3282c3b 2023-04-16 00:56:34,438 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=1289, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp/A/4d5fd665263245c1b04384fb2d80f5e3 2023-04-16 00:56:34,449 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=1289 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp/B/5b5b4eb2e1ba434b9d99bf5de9824275 2023-04-16 00:56:34,460 DEBUG [Thread-4138] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:56:34,461 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:56:34,462 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-16 00:56:34,463 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-16 00:56:34,463 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-16 00:56:34,463 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-16 00:56:34,463 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:56:34,463 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-16 00:56:34,463 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-16 00:56:34,464 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:34,464 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:34,465 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:56:34,465 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:34,465 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:56:34,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:56:34,465 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:34,465 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-16 00:56:34,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:34,466 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:34,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-16 00:56:34,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:34,466 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-16 00:56:34,466 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-16 00:56:34,466 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-16 00:56:34,466 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-16 00:56:34,466 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-16 00:56:34,466 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-16 00:56:34,467 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-16 00:56:34,467 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-16 00:56:34,467 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. 2023-04-16 00:56:34,467 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. started... 2023-04-16 00:56:34,467 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. as already flushing 2023-04-16 00:56:34,467 DEBUG [rs(jenkins-hbase4.apache.org,34975,1681606419217)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. 2023-04-16 00:56:34,467 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-16 00:56:34,467 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-16 00:56:34,467 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:34,467 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-16 00:56:34,467 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-16 00:56:34,468 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:34,469 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:34,469 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:34,469 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-16 00:56:34,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:34,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:34,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:34,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:34,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:34,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:34,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:34,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:34,470 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34975,1681606419217' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-16 00:56:34,470 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-16 00:56:34,470 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:34,470 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2b90b955[Count = 0] remaining members to acquire global barrier 2023-04-16 00:56:34,471 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:34,471 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:34,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:34,471 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-16 00:56:34,471 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-16 00:56:34,471 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,34975,1681606419217' in zk 2023-04-16 00:56:34,472 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:34,472 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-16 00:56:34,474 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:34,474 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-16 00:56:34,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:34,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:34,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:34,474 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:56:34,474 DEBUG [member: 'jenkins-hbase4.apache.org,34975,1681606419217' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-16 00:56:34,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:34,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:34,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:34,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:34,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:34,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:34,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:34,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,34975,1681606419217': 2023-04-16 00:56:34,477 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34975,1681606419217' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-16 00:56:34,477 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-16 00:56:34,477 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-16 00:56:34,477 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-16 00:56:34,477 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-16 00:56:34,477 INFO [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-16 00:56:34,478 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:34,478 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:34,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:34,478 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:34,478 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:56:34,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:34,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-16 00:56:34,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-16 00:56:34,479 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:56:34,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:56:34,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:34,479 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:34,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-16 00:56:34,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:34,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-16 00:56:34,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:34,480 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:34,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:34,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-16 00:56:34,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-16 00:56:34,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:34,482 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-16 00:56:34,482 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-16 00:56:34,482 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-16 00:56:34,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-16 00:56:34,482 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-16 00:56:34,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-16 00:56:34,483 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:34,483 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:34,483 DEBUG [(jenkins-hbase4.apache.org,38795,1681606417971)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-16 00:56:34,483 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-16 00:56:34,483 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-16 00:56:34,483 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:34,483 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:34,483 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-16 00:56:34,483 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-16 00:56:34,484 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-16 00:56:34,484 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-16 00:56:34,484 DEBUG [Thread-4138] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-16 00:56:34,484 DEBUG [Thread-4138] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-16 00:56:34,525 DEBUG [Thread-4147] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3d765193 to 127.0.0.1:53150 2023-04-16 00:56:34,525 DEBUG [Thread-4147] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-16 00:56:34,526 DEBUG [Thread-4143] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2c53bf96 to 127.0.0.1:53150 2023-04-16 00:56:34,526 DEBUG [Thread-4143] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-16 00:56:34,527 DEBUG [Thread-4139] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x544a4215 to 127.0.0.1:53150 2023-04-16 00:56:34,527 DEBUG [Thread-4139] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-16 00:56:34,528 DEBUG [Thread-4141] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x25baee3e to 127.0.0.1:53150 2023-04-16 00:56:34,528 DEBUG [Thread-4141] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-16 00:56:34,530 DEBUG [Thread-4145] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3b59bdcf to 127.0.0.1:53150 2023-04-16 00:56:34,530 DEBUG [Thread-4145] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-16 00:56:34,584 DEBUG [Thread-4138] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-16 00:56:34,585 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-16 00:56:34,651 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:34,651 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 410 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44548 deadline: 1681606654651, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:34,665 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:34,666 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 378 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44554 deadline: 1681606654665, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:34,667 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:34,667 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:34,667 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34975] ipc.CallRunner(144): callId: 305 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44564 deadline: 1681606654667, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:34,667 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 324 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44556 deadline: 1681606654667, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:34,668 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-16 00:56:34,668 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] ipc.CallRunner(144): callId: 347 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:44562 deadline: 1681606654668, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:34,857 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=1289 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp/C/261635fdb2db4c0e89cf7fb6264e69fe 2023-04-16 00:56:34,860 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp/A/4d5fd665263245c1b04384fb2d80f5e3 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A/4d5fd665263245c1b04384fb2d80f5e3 2023-04-16 00:56:34,863 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A/4d5fd665263245c1b04384fb2d80f5e3, entries=250, sequenceid=1289, filesize=39.2 K 2023-04-16 00:56:34,863 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp/B/5b5b4eb2e1ba434b9d99bf5de9824275 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/5b5b4eb2e1ba434b9d99bf5de9824275 2023-04-16 00:56:34,866 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/5b5b4eb2e1ba434b9d99bf5de9824275, entries=150, sequenceid=1289, filesize=11.9 K 2023-04-16 00:56:34,866 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp/C/261635fdb2db4c0e89cf7fb6264e69fe as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/261635fdb2db4c0e89cf7fb6264e69fe 2023-04-16 00:56:34,868 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/261635fdb2db4c0e89cf7fb6264e69fe, entries=150, sequenceid=1289, filesize=11.9 K 2023-04-16 00:56:34,869 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for d538367277459b70f4a06235dfaacfec in 851ms, sequenceid=1289, compaction requested=true 2023-04-16 00:56:34,869 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for d538367277459b70f4a06235dfaacfec: 2023-04-16 00:56:34,869 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 00:56:34,869 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 00:56:34,869 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-16 00:56:34,869 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-16 00:56:34,869 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 00:56:34,870 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 93754 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-16 00:56:34,870 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 38124 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-16 00:56:34,870 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HStore(1912): d538367277459b70f4a06235dfaacfec/A is initiating minor compaction (all files) 2023-04-16 00:56:34,870 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HStore(1912): d538367277459b70f4a06235dfaacfec/B is initiating minor compaction (all files) 2023-04-16 00:56:34,870 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of d538367277459b70f4a06235dfaacfec/A in TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. 2023-04-16 00:56:34,870 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HRegion(2259): Starting compaction of d538367277459b70f4a06235dfaacfec/B in TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. 2023-04-16 00:56:34,870 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A/91feb7f67cc64dabbd8922d9cf8d3c5a, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A/3532e4998ae246be82011ecdb3cb9f94, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A/4d5fd665263245c1b04384fb2d80f5e3] into tmpdir=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp, totalSize=91.6 K 2023-04-16 00:56:34,870 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/4a9ae918db844b738da120a5a9eef156, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/f8a86b4f975d46ce9c525645e6a5aeb5, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/5b5b4eb2e1ba434b9d99bf5de9824275] into tmpdir=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp, totalSize=37.2 K 2023-04-16 00:56:34,871 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.Compactor(207): Compacting 91feb7f67cc64dabbd8922d9cf8d3c5a, keycount=150, bloomtype=ROW, size=26.9 K, encoding=NONE, compression=NONE, seqNum=1238, earliestPutTs=1681606593302 2023-04-16 00:56:34,871 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting 4a9ae918db844b738da120a5a9eef156, keycount=150, bloomtype=ROW, size=13.4 K, encoding=NONE, compression=NONE, seqNum=1238, earliestPutTs=1681606593302 2023-04-16 00:56:34,871 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.Compactor(207): Compacting 3532e4998ae246be82011ecdb3cb9f94, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=1277, earliestPutTs=1681606593366 2023-04-16 00:56:34,871 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting f8a86b4f975d46ce9c525645e6a5aeb5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1277, earliestPutTs=1681606593366 2023-04-16 00:56:34,871 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] compactions.Compactor(207): Compacting 4d5fd665263245c1b04384fb2d80f5e3, keycount=250, bloomtype=ROW, size=39.2 K, encoding=NONE, compression=NONE, seqNum=1289, earliestPutTs=1681606593998 2023-04-16 00:56:34,871 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting 5b5b4eb2e1ba434b9d99bf5de9824275, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1289, earliestPutTs=1681606594014 2023-04-16 00:56:34,875 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] throttle.PressureAwareThroughputController(145): d538367277459b70f4a06235dfaacfec#B#compaction#702 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-16 00:56:34,878 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] throttle.PressureAwareThroughputController(145): d538367277459b70f4a06235dfaacfec#A#compaction#701 average throughput is 6.57 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-16 00:56:34,882 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp/B/8985806958fd4bb4a0757e4e45134bb1 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/8985806958fd4bb4a0757e4e45134bb1 2023-04-16 00:56:34,886 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in d538367277459b70f4a06235dfaacfec/B of d538367277459b70f4a06235dfaacfec into 8985806958fd4bb4a0757e4e45134bb1(size=13.5 K), total size for store is 13.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 00:56:34,886 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for d538367277459b70f4a06235dfaacfec: 2023-04-16 00:56:34,886 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec., storeName=d538367277459b70f4a06235dfaacfec/B, priority=13, startTime=1681606594869; duration=0sec 2023-04-16 00:56:34,886 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-16 00:56:34,886 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-16 00:56:34,887 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 38124 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-16 00:56:34,887 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HStore(1912): d538367277459b70f4a06235dfaacfec/C is initiating minor compaction (all files) 2023-04-16 00:56:34,887 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HRegion(2259): Starting compaction of d538367277459b70f4a06235dfaacfec/C in TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. 2023-04-16 00:56:34,887 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/4492b0b9e8c64fef9380ae8a5f8af5ad, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/ea3f2d4b75654f28aeca92dc38cec363, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/261635fdb2db4c0e89cf7fb6264e69fe] into tmpdir=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp, totalSize=37.2 K 2023-04-16 00:56:34,887 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting 4492b0b9e8c64fef9380ae8a5f8af5ad, keycount=150, bloomtype=ROW, size=13.4 K, encoding=NONE, compression=NONE, seqNum=1238, earliestPutTs=1681606593302 2023-04-16 00:56:34,887 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting ea3f2d4b75654f28aeca92dc38cec363, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1277, earliestPutTs=1681606593366 2023-04-16 00:56:34,888 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] compactions.Compactor(207): Compacting 261635fdb2db4c0e89cf7fb6264e69fe, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1289, earliestPutTs=1681606594014 2023-04-16 00:56:34,892 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] throttle.PressureAwareThroughputController(145): d538367277459b70f4a06235dfaacfec#C#compaction#703 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-16 00:56:34,897 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp/C/2c49b07d3f5a414581ebcc21ffcca00d as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/2c49b07d3f5a414581ebcc21ffcca00d 2023-04-16 00:56:34,900 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in d538367277459b70f4a06235dfaacfec/C of d538367277459b70f4a06235dfaacfec into 2c49b07d3f5a414581ebcc21ffcca00d(size=13.5 K), total size for store is 13.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 00:56:34,900 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for d538367277459b70f4a06235dfaacfec: 2023-04-16 00:56:34,900 INFO [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec., storeName=d538367277459b70f4a06235dfaacfec/C, priority=13, startTime=1681606594869; duration=0sec 2023-04-16 00:56:34,900 DEBUG [RS:0;jenkins-hbase4:34975-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 00:56:35,155 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34975] regionserver.HRegion(9158): Flush requested on d538367277459b70f4a06235dfaacfec 2023-04-16 00:56:35,155 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing d538367277459b70f4a06235dfaacfec 3/3 column families, dataSize=134.18 KB heapSize=380.30 KB 2023-04-16 00:56:35,156 DEBUG [Thread-4136] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4db22237 to 127.0.0.1:53150 2023-04-16 00:56:35,156 DEBUG [Thread-4136] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-16 00:56:35,156 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d538367277459b70f4a06235dfaacfec, store=A 2023-04-16 00:56:35,156 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 00:56:35,156 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d538367277459b70f4a06235dfaacfec, store=B 2023-04-16 00:56:35,156 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 00:56:35,156 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d538367277459b70f4a06235dfaacfec, store=C 2023-04-16 00:56:35,156 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-16 00:56:35,169 DEBUG [Thread-4134] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5f9e8654 to 127.0.0.1:53150 2023-04-16 00:56:35,169 DEBUG [Thread-4134] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-16 00:56:35,170 DEBUG [Thread-4132] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6f06dd50 to 127.0.0.1:53150 2023-04-16 00:56:35,170 DEBUG [Thread-4132] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-16 00:56:35,172 DEBUG [Thread-4128] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x470742aa to 127.0.0.1:53150 2023-04-16 00:56:35,172 DEBUG [Thread-4128] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-16 00:56:35,173 DEBUG [Thread-4130] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1e3503b4 to 127.0.0.1:53150 2023-04-16 00:56:35,173 DEBUG [Thread-4130] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-16 00:56:35,173 INFO [Listener at localhost/33477] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-16 00:56:35,173 INFO [Listener at localhost/33477] hbase.AcidGuaranteesTestTool(394): wrote 217 2023-04-16 00:56:35,173 INFO [Listener at localhost/33477] hbase.AcidGuaranteesTestTool(394): wrote 218 2023-04-16 00:56:35,173 INFO [Listener at localhost/33477] hbase.AcidGuaranteesTestTool(394): wrote 253 2023-04-16 00:56:35,173 INFO [Listener at localhost/33477] hbase.AcidGuaranteesTestTool(394): wrote 196 2023-04-16 00:56:35,173 INFO [Listener at localhost/33477] hbase.AcidGuaranteesTestTool(394): wrote 285 2023-04-16 00:56:35,173 INFO [Listener at localhost/33477] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-16 00:56:35,173 INFO [Listener at localhost/33477] hbase.AcidGuaranteesTestTool(398): read 4216 2023-04-16 00:56:35,173 INFO [Listener at localhost/33477] hbase.AcidGuaranteesTestTool(398): read 4279 2023-04-16 00:56:35,173 INFO [Listener at localhost/33477] hbase.AcidGuaranteesTestTool(398): read 4217 2023-04-16 00:56:35,173 INFO [Listener at localhost/33477] hbase.AcidGuaranteesTestTool(398): read 4239 2023-04-16 00:56:35,173 INFO [Listener at localhost/33477] hbase.AcidGuaranteesTestTool(398): read 4307 2023-04-16 00:56:35,173 INFO [Listener at localhost/33477] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-16 00:56:35,173 INFO [Listener at localhost/33477] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-16 00:56:35,173 DEBUG [Listener at localhost/33477] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x50529bf3 to 127.0.0.1:53150 2023-04-16 00:56:35,174 DEBUG [Listener at localhost/33477] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-16 00:56:35,174 INFO [Listener at localhost/33477] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-16 00:56:35,174 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable TestAcidGuarantees 2023-04-16 00:56:35,175 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure2.ProcedureExecutor(1029): Stored pid=62, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-16 00:56:35,177 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-16 00:56:35,177 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681606595177"}]},"ts":"1681606595177"} 2023-04-16 00:56:35,178 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-16 00:56:35,275 INFO [PEWorker-5] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-16 00:56:35,276 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=63, ppid=62, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=d538367277459b70f4a06235dfaacfec, UNASSIGN}] 2023-04-16 00:56:35,277 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=63, ppid=62, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=d538367277459b70f4a06235dfaacfec, UNASSIGN 2023-04-16 00:56:35,278 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-16 00:56:35,278 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=63 updating hbase:meta row=d538367277459b70f4a06235dfaacfec, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:35,278 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681606595278"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681606595278"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681606595278"}]},"ts":"1681606595278"} 2023-04-16 00:56:35,279 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=64, ppid=63, state=RUNNABLE; CloseRegionProcedure d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217}] 2023-04-16 00:56:35,291 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp/A/ef9f5ecad71e408bba6d3eda91b99c1f as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A/ef9f5ecad71e408bba6d3eda91b99c1f 2023-04-16 00:56:35,294 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in d538367277459b70f4a06235dfaacfec/A of d538367277459b70f4a06235dfaacfec into ef9f5ecad71e408bba6d3eda91b99c1f(size=27.0 K), total size for store is 27.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-16 00:56:35,294 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for d538367277459b70f4a06235dfaacfec: 2023-04-16 00:56:35,294 INFO [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec., storeName=d538367277459b70f4a06235dfaacfec/A, priority=13, startTime=1681606594869; duration=0sec 2023-04-16 00:56:35,295 DEBUG [RS:0;jenkins-hbase4:34975-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-16 00:56:35,430 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close d538367277459b70f4a06235dfaacfec 2023-04-16 00:56:35,431 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing d538367277459b70f4a06235dfaacfec, disabling compactions & flushes 2023-04-16 00:56:35,431 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1858): waiting for 0 compactions & cache flush to complete for region TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. 2023-04-16 00:56:35,479 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-16 00:56:35,566 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230416b5f3c80037c8436db353ac66fbabdb16 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230416b5f3c80037c8436db353ac66fbabdb16 2023-04-16 00:56:35,780 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-16 00:56:35,970 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=1327, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp/A/a3070582caca409c81b33e41f756f35c 2023-04-16 00:56:36,281 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-16 00:56:36,378 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=1327 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp/B/70fe19ce2dc84ba6b40a73c2f0780b84 2023-04-16 00:56:36,787 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=1327 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp/C/460c0e2776fc4cc9a516ed393b868a2d 2023-04-16 00:56:36,790 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp/A/a3070582caca409c81b33e41f756f35c as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A/a3070582caca409c81b33e41f756f35c 2023-04-16 00:56:36,792 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A/a3070582caca409c81b33e41f756f35c, entries=150, sequenceid=1327, filesize=25.4 K 2023-04-16 00:56:36,793 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp/B/70fe19ce2dc84ba6b40a73c2f0780b84 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/70fe19ce2dc84ba6b40a73c2f0780b84 2023-04-16 00:56:36,796 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/70fe19ce2dc84ba6b40a73c2f0780b84, entries=150, sequenceid=1327, filesize=11.9 K 2023-04-16 00:56:36,796 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp/C/460c0e2776fc4cc9a516ed393b868a2d as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/460c0e2776fc4cc9a516ed393b868a2d 2023-04-16 00:56:36,799 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/460c0e2776fc4cc9a516ed393b868a2d, entries=150, sequenceid=1327, filesize=11.9 K 2023-04-16 00:56:36,799 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~380.25 KB/389376, currentSize=26.84 KB/27480 for d538367277459b70f4a06235dfaacfec in 1644ms, sequenceid=1327, compaction requested=false 2023-04-16 00:56:36,800 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for d538367277459b70f4a06235dfaacfec: 2023-04-16 00:56:36,800 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. 2023-04-16 00:56:36,800 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. 2023-04-16 00:56:36,800 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. after waiting 0 ms 2023-04-16 00:56:36,800 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. 2023-04-16 00:56:36,800 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing d538367277459b70f4a06235dfaacfec 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-04-16 00:56:36,800 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d538367277459b70f4a06235dfaacfec, store=A 2023-04-16 00:56:36,800 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:56:36,800 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d538367277459b70f4a06235dfaacfec, store=B 2023-04-16 00:56:36,800 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:56:36,800 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d538367277459b70f4a06235dfaacfec, store=C 2023-04-16 00:56:36,800 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-16 00:56:37,209 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230416572fffa0922444028877c008cb866454 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230416572fffa0922444028877c008cb866454 2023-04-16 00:56:37,282 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-16 00:56:37,614 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=1335, memsize=8.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp/A/42eab723fbfc4befb3a19c59e08d0032 2023-04-16 00:56:38,022 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=1335 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp/B/b8e276a847b44ebc8c86de137ebffb11 2023-04-16 00:56:38,430 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=1335 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp/C/c54e5cfc0a824d31ad1f163d399b71ac 2023-04-16 00:56:38,433 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp/A/42eab723fbfc4befb3a19c59e08d0032 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A/42eab723fbfc4befb3a19c59e08d0032 2023-04-16 00:56:38,436 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A/42eab723fbfc4befb3a19c59e08d0032, entries=150, sequenceid=1335, filesize=25.4 K 2023-04-16 00:56:38,436 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp/B/b8e276a847b44ebc8c86de137ebffb11 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/b8e276a847b44ebc8c86de137ebffb11 2023-04-16 00:56:38,439 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/b8e276a847b44ebc8c86de137ebffb11, entries=150, sequenceid=1335, filesize=11.9 K 2023-04-16 00:56:38,440 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/.tmp/C/c54e5cfc0a824d31ad1f163d399b71ac as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/c54e5cfc0a824d31ad1f163d399b71ac 2023-04-16 00:56:38,442 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/c54e5cfc0a824d31ad1f163d399b71ac, entries=150, sequenceid=1335, filesize=11.9 K 2023-04-16 00:56:38,443 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=0 B/0 for d538367277459b70f4a06235dfaacfec in 1643ms, sequenceid=1335, compaction requested=true 2023-04-16 00:56:38,476 DEBUG [StoreCloser-TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A/abd5cdebfeef4d6e8e910c771a0c80aa, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A/9f993bd920d148539f837ee17dae4620, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A/3d98fe119a4c4b74818836f84975adfb, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A/381fa88c7f654cbf9ad13e96d34818b1, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A/28c735ea22c94a079f00471488c02e8c, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A/c09f3418d25c46479eaed2c8c09d2740, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A/cb9acde7623542dc9f35614296829d70, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A/a43dd4d44481416c85d473f9b3e6639f, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A/25122e76a3ac47b8accde73fc4baa0c1, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A/1d92c3f646be4e918c53aa9745ef7bb5, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A/21025b18a78649a88ca998e1d8ebae26, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A/3947ddc746d24d7e80c706bc115db731, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A/d6467e9fff6b4b00b6f7f74a31c43aee, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A/cd94a09e5f6e4122ac967486e097084c, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A/e5ac83fbb7684a0ca28b839dfff5231e, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A/cdd41307649a455fbb20b175cacaec3c, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A/b2f263a74dc3430bbca4c42cdaccc326, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A/6a2bcf616ac34837a1b83dcfc086d5fe, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A/74ed64aee6634e1db91e1b19e0f17a62, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A/fb09760c8d434f05b086841c634310f8, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A/fe26a1dd9f1a47328a6b6173cc96d91b, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A/da12a06b31d441d599d829b0ff669323, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A/22f90baaf5e44bbd86a35ea3cd725c23, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A/7582b7a5f2454c368c0ad8a4637a6015, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A/19ac16e53a404a87a89e9f69b765373f, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A/f7aa501a91d443f9a493a7e3a88884b2, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A/fc4225da4ed44d7a9492b896f4e9001b, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A/22db59fae0354e219d8e9c6da3fd024b, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A/d49c5ed4e0ff46a09e6ccf4ae0db5059, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A/8317419ec6ed413eb9152bb51d493d86, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A/e242a6eb9be84c319cd7cd35d15f976f, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A/10cdf46636424ee38fafe671851b987e, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A/a11321a458434e23b0e455ab0c787fa5, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A/1122a71abf6a45af95f9c3a6185f56fe, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A/a5afc425ea0b4a36acfd05a67e740914, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A/d97367c08cde484ba1badf53f5a0366f, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A/67fe03bbff1c41b6adeccd6cee68c9e2, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A/b18b0a2fbb8640388db4ee137f17466e, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A/682a8b9d78f9467d8142c539b349897e, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A/1d0ac54425864bdfa58b8d40561b8e29, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A/2f08f67213f7492eac8cc1240f416b9e, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A/5c5ede44047045a689058706ef154979, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A/175324f03802417cb8a838e0046c05a4, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A/70ce4b2231554814bb0fdde6d896520c, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A/f16aacb5158b4551b6b5e6903ff792b3, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A/91feb7f67cc64dabbd8922d9cf8d3c5a, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A/3532e4998ae246be82011ecdb3cb9f94, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A/4d5fd665263245c1b04384fb2d80f5e3] to archive 2023-04-16 00:56:38,477 DEBUG [StoreCloser-TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-16 00:56:38,478 DEBUG [StoreCloser-TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A/abd5cdebfeef4d6e8e910c771a0c80aa to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A/abd5cdebfeef4d6e8e910c771a0c80aa 2023-04-16 00:56:38,479 DEBUG [StoreCloser-TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A/9f993bd920d148539f837ee17dae4620 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A/9f993bd920d148539f837ee17dae4620 2023-04-16 00:56:38,480 DEBUG [StoreCloser-TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A/3d98fe119a4c4b74818836f84975adfb to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A/3d98fe119a4c4b74818836f84975adfb 2023-04-16 00:56:38,481 DEBUG [StoreCloser-TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A/381fa88c7f654cbf9ad13e96d34818b1 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A/381fa88c7f654cbf9ad13e96d34818b1 2023-04-16 00:56:38,482 DEBUG [StoreCloser-TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A/28c735ea22c94a079f00471488c02e8c to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A/28c735ea22c94a079f00471488c02e8c 2023-04-16 00:56:38,483 DEBUG [StoreCloser-TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A/c09f3418d25c46479eaed2c8c09d2740 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A/c09f3418d25c46479eaed2c8c09d2740 2023-04-16 00:56:38,484 DEBUG [StoreCloser-TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A/cb9acde7623542dc9f35614296829d70 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A/cb9acde7623542dc9f35614296829d70 2023-04-16 00:56:38,485 DEBUG [StoreCloser-TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A/a43dd4d44481416c85d473f9b3e6639f to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A/a43dd4d44481416c85d473f9b3e6639f 2023-04-16 00:56:38,486 DEBUG [StoreCloser-TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A/25122e76a3ac47b8accde73fc4baa0c1 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A/25122e76a3ac47b8accde73fc4baa0c1 2023-04-16 00:56:38,486 DEBUG [StoreCloser-TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A/1d92c3f646be4e918c53aa9745ef7bb5 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A/1d92c3f646be4e918c53aa9745ef7bb5 2023-04-16 00:56:38,487 DEBUG [StoreCloser-TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A/21025b18a78649a88ca998e1d8ebae26 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A/21025b18a78649a88ca998e1d8ebae26 2023-04-16 00:56:38,488 DEBUG [StoreCloser-TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A/3947ddc746d24d7e80c706bc115db731 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A/3947ddc746d24d7e80c706bc115db731 2023-04-16 00:56:38,489 DEBUG [StoreCloser-TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A/d6467e9fff6b4b00b6f7f74a31c43aee to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A/d6467e9fff6b4b00b6f7f74a31c43aee 2023-04-16 00:56:38,490 DEBUG [StoreCloser-TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A/cd94a09e5f6e4122ac967486e097084c to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A/cd94a09e5f6e4122ac967486e097084c 2023-04-16 00:56:38,491 DEBUG [StoreCloser-TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A/e5ac83fbb7684a0ca28b839dfff5231e to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A/e5ac83fbb7684a0ca28b839dfff5231e 2023-04-16 00:56:38,491 DEBUG [StoreCloser-TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A/cdd41307649a455fbb20b175cacaec3c to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A/cdd41307649a455fbb20b175cacaec3c 2023-04-16 00:56:38,492 DEBUG [StoreCloser-TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A/b2f263a74dc3430bbca4c42cdaccc326 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A/b2f263a74dc3430bbca4c42cdaccc326 2023-04-16 00:56:38,493 DEBUG [StoreCloser-TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A/6a2bcf616ac34837a1b83dcfc086d5fe to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A/6a2bcf616ac34837a1b83dcfc086d5fe 2023-04-16 00:56:38,494 DEBUG [StoreCloser-TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A/74ed64aee6634e1db91e1b19e0f17a62 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A/74ed64aee6634e1db91e1b19e0f17a62 2023-04-16 00:56:38,495 DEBUG [StoreCloser-TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A/fb09760c8d434f05b086841c634310f8 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A/fb09760c8d434f05b086841c634310f8 2023-04-16 00:56:38,496 DEBUG [StoreCloser-TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A/fe26a1dd9f1a47328a6b6173cc96d91b to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A/fe26a1dd9f1a47328a6b6173cc96d91b 2023-04-16 00:56:38,497 DEBUG [StoreCloser-TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A/da12a06b31d441d599d829b0ff669323 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A/da12a06b31d441d599d829b0ff669323 2023-04-16 00:56:38,498 DEBUG [StoreCloser-TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A/22f90baaf5e44bbd86a35ea3cd725c23 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A/22f90baaf5e44bbd86a35ea3cd725c23 2023-04-16 00:56:38,498 DEBUG [StoreCloser-TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A/7582b7a5f2454c368c0ad8a4637a6015 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A/7582b7a5f2454c368c0ad8a4637a6015 2023-04-16 00:56:38,499 DEBUG [StoreCloser-TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A/19ac16e53a404a87a89e9f69b765373f to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A/19ac16e53a404a87a89e9f69b765373f 2023-04-16 00:56:38,500 DEBUG [StoreCloser-TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A/f7aa501a91d443f9a493a7e3a88884b2 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A/f7aa501a91d443f9a493a7e3a88884b2 2023-04-16 00:56:38,501 DEBUG [StoreCloser-TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A/fc4225da4ed44d7a9492b896f4e9001b to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A/fc4225da4ed44d7a9492b896f4e9001b 2023-04-16 00:56:38,502 DEBUG [StoreCloser-TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A/22db59fae0354e219d8e9c6da3fd024b to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A/22db59fae0354e219d8e9c6da3fd024b 2023-04-16 00:56:38,502 DEBUG [StoreCloser-TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A/d49c5ed4e0ff46a09e6ccf4ae0db5059 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A/d49c5ed4e0ff46a09e6ccf4ae0db5059 2023-04-16 00:56:38,503 DEBUG [StoreCloser-TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A/8317419ec6ed413eb9152bb51d493d86 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A/8317419ec6ed413eb9152bb51d493d86 2023-04-16 00:56:38,504 DEBUG [StoreCloser-TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A/e242a6eb9be84c319cd7cd35d15f976f to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A/e242a6eb9be84c319cd7cd35d15f976f 2023-04-16 00:56:38,505 DEBUG [StoreCloser-TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A/10cdf46636424ee38fafe671851b987e to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A/10cdf46636424ee38fafe671851b987e 2023-04-16 00:56:38,506 DEBUG [StoreCloser-TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A/a11321a458434e23b0e455ab0c787fa5 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A/a11321a458434e23b0e455ab0c787fa5 2023-04-16 00:56:38,506 DEBUG [StoreCloser-TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A/1122a71abf6a45af95f9c3a6185f56fe to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A/1122a71abf6a45af95f9c3a6185f56fe 2023-04-16 00:56:38,507 DEBUG [StoreCloser-TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A/a5afc425ea0b4a36acfd05a67e740914 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A/a5afc425ea0b4a36acfd05a67e740914 2023-04-16 00:56:38,508 DEBUG [StoreCloser-TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A/d97367c08cde484ba1badf53f5a0366f to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A/d97367c08cde484ba1badf53f5a0366f 2023-04-16 00:56:38,509 DEBUG [StoreCloser-TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A/67fe03bbff1c41b6adeccd6cee68c9e2 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A/67fe03bbff1c41b6adeccd6cee68c9e2 2023-04-16 00:56:38,510 DEBUG [StoreCloser-TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A/b18b0a2fbb8640388db4ee137f17466e to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A/b18b0a2fbb8640388db4ee137f17466e 2023-04-16 00:56:38,511 DEBUG [StoreCloser-TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A/682a8b9d78f9467d8142c539b349897e to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A/682a8b9d78f9467d8142c539b349897e 2023-04-16 00:56:38,512 DEBUG [StoreCloser-TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A/1d0ac54425864bdfa58b8d40561b8e29 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A/1d0ac54425864bdfa58b8d40561b8e29 2023-04-16 00:56:38,513 DEBUG [StoreCloser-TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A/2f08f67213f7492eac8cc1240f416b9e to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A/2f08f67213f7492eac8cc1240f416b9e 2023-04-16 00:56:38,513 DEBUG [StoreCloser-TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A/5c5ede44047045a689058706ef154979 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A/5c5ede44047045a689058706ef154979 2023-04-16 00:56:38,514 DEBUG [StoreCloser-TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A/175324f03802417cb8a838e0046c05a4 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A/175324f03802417cb8a838e0046c05a4 2023-04-16 00:56:38,515 DEBUG [StoreCloser-TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A/70ce4b2231554814bb0fdde6d896520c to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A/70ce4b2231554814bb0fdde6d896520c 2023-04-16 00:56:38,515 DEBUG [StoreCloser-TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A/f16aacb5158b4551b6b5e6903ff792b3 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A/f16aacb5158b4551b6b5e6903ff792b3 2023-04-16 00:56:38,516 DEBUG [StoreCloser-TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A/91feb7f67cc64dabbd8922d9cf8d3c5a to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A/91feb7f67cc64dabbd8922d9cf8d3c5a 2023-04-16 00:56:38,517 DEBUG [StoreCloser-TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A/3532e4998ae246be82011ecdb3cb9f94 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A/3532e4998ae246be82011ecdb3cb9f94 2023-04-16 00:56:38,518 DEBUG [StoreCloser-TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A/4d5fd665263245c1b04384fb2d80f5e3 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A/4d5fd665263245c1b04384fb2d80f5e3 2023-04-16 00:56:38,523 DEBUG [StoreCloser-TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/df0e86e715234f7aa1b3cb278f41ed4c, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/621929dbbab54631aa24fa3dfc6566b4, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/d0f033214214487fa0a5f40833b417f6, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/a148cb76b0584b23afed65b3b63fd481, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/be370d9d99b142a2a9d617304ddc164a, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/697a03eb36f64a8ba6a8436f67894577, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/5662ac01a6d24740bbfcef0230bcb736, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/b931b8c998cf402d837c49a53305f3ad, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/7b6da3ebcd154ab7af07a6697dd4a150, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/ee3926f1cb034130a58d1315408ab2f1, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/4e5fb4ddba9c48bd9313d6d4be4dcd3c, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/2ddc315b0b184c4fb52a9135c43dc968, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/eee4a4e7ffc1455fb34c55e7796870bb, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/2c25715a76ec4aa9bbe7cdaa21f15cf5, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/f242a0f93a004e8e87a5b468b65420cd, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/98d55edf116f4da28ce9d4768d606e00, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/08ad153cb7df419da541f7ba065dd7e3, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/e45986fdbe5b4015af9ea0599530c7c4, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/9cc616814cdc428a80348c74c031785b, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/6140e76fab054aa4aa6ca6ceb9f73fba, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/000d29b1df9e4c9f8b4fd8726653f465, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/e5965a0411c44477b420d28bd13dd872, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/f62174af8c2047a3b8ad29743a771078, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/0454d7b6b16e44c79e571242d0b3a390, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/c5e34cd11dd64f82a0c80a478d39dc46, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/98f28f3e808b4fdabf1fa12b5c18e302, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/aa952594b8ee4cbd8d7653fcbb8da493, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/10af9a31afdd44b2bb4c51e276db3426, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/e917078e0ff64ebea0587c7db31a293a, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/7ba5a16a30174fc09fe5b6e4ae542936, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/c54e596b6b774e48b4ac80b98342a4fd, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/c8f6e4654d98421c82dac933bc788866, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/c5d3816f8b4d4d149e50042d81e8ba7b, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/e21e02618d174623b613a8d1163f365f, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/36121c3b7ff04cf09f73986d1ff097e3, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/1b14d0512fb54b2ca3808faeebf965a6, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/2e98666f01c24e69a099e8eee156ee6e, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/c95993956ef44794ab016c69f0c94946, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/007c353b19c74958a6b7f5cc4c51fece, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/f59a5489aa924fe2a05e495dc0ee4e2c, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/703ddd1cbced43b287286c523fcea3fa, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/a1676505a0984a60834a0f721fd8c557, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/78e3e91f3b5e4ea997d8229031a4ed0c, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/5455b9b925c2464aa6425061dea32d4a, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/a17dd28a2c0243fbba5a7c589e5dce26, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/4a9ae918db844b738da120a5a9eef156, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/fb0afb1309664563a14a31e108cd473b, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/f8a86b4f975d46ce9c525645e6a5aeb5, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/5b5b4eb2e1ba434b9d99bf5de9824275] to archive 2023-04-16 00:56:38,523 DEBUG [StoreCloser-TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-16 00:56:38,525 DEBUG [StoreCloser-TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/df0e86e715234f7aa1b3cb278f41ed4c to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/df0e86e715234f7aa1b3cb278f41ed4c 2023-04-16 00:56:38,525 DEBUG [StoreCloser-TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/621929dbbab54631aa24fa3dfc6566b4 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/621929dbbab54631aa24fa3dfc6566b4 2023-04-16 00:56:38,526 DEBUG [StoreCloser-TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/d0f033214214487fa0a5f40833b417f6 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/d0f033214214487fa0a5f40833b417f6 2023-04-16 00:56:38,527 DEBUG [StoreCloser-TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/a148cb76b0584b23afed65b3b63fd481 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/a148cb76b0584b23afed65b3b63fd481 2023-04-16 00:56:38,528 DEBUG [StoreCloser-TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/be370d9d99b142a2a9d617304ddc164a to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/be370d9d99b142a2a9d617304ddc164a 2023-04-16 00:56:38,528 DEBUG [StoreCloser-TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/697a03eb36f64a8ba6a8436f67894577 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/697a03eb36f64a8ba6a8436f67894577 2023-04-16 00:56:38,529 DEBUG [StoreCloser-TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/5662ac01a6d24740bbfcef0230bcb736 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/5662ac01a6d24740bbfcef0230bcb736 2023-04-16 00:56:38,530 DEBUG [StoreCloser-TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/b931b8c998cf402d837c49a53305f3ad to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/b931b8c998cf402d837c49a53305f3ad 2023-04-16 00:56:38,531 DEBUG [StoreCloser-TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/7b6da3ebcd154ab7af07a6697dd4a150 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/7b6da3ebcd154ab7af07a6697dd4a150 2023-04-16 00:56:38,532 DEBUG [StoreCloser-TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/ee3926f1cb034130a58d1315408ab2f1 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/ee3926f1cb034130a58d1315408ab2f1 2023-04-16 00:56:38,532 DEBUG [StoreCloser-TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/4e5fb4ddba9c48bd9313d6d4be4dcd3c to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/4e5fb4ddba9c48bd9313d6d4be4dcd3c 2023-04-16 00:56:38,533 DEBUG [StoreCloser-TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/2ddc315b0b184c4fb52a9135c43dc968 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/2ddc315b0b184c4fb52a9135c43dc968 2023-04-16 00:56:38,534 DEBUG [StoreCloser-TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/eee4a4e7ffc1455fb34c55e7796870bb to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/eee4a4e7ffc1455fb34c55e7796870bb 2023-04-16 00:56:38,535 DEBUG [StoreCloser-TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/2c25715a76ec4aa9bbe7cdaa21f15cf5 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/2c25715a76ec4aa9bbe7cdaa21f15cf5 2023-04-16 00:56:38,536 DEBUG [StoreCloser-TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/f242a0f93a004e8e87a5b468b65420cd to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/f242a0f93a004e8e87a5b468b65420cd 2023-04-16 00:56:38,537 DEBUG [StoreCloser-TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/98d55edf116f4da28ce9d4768d606e00 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/98d55edf116f4da28ce9d4768d606e00 2023-04-16 00:56:38,538 DEBUG [StoreCloser-TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/08ad153cb7df419da541f7ba065dd7e3 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/08ad153cb7df419da541f7ba065dd7e3 2023-04-16 00:56:38,538 DEBUG [StoreCloser-TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/e45986fdbe5b4015af9ea0599530c7c4 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/e45986fdbe5b4015af9ea0599530c7c4 2023-04-16 00:56:38,539 DEBUG [StoreCloser-TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/9cc616814cdc428a80348c74c031785b to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/9cc616814cdc428a80348c74c031785b 2023-04-16 00:56:38,540 DEBUG [StoreCloser-TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/6140e76fab054aa4aa6ca6ceb9f73fba to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/6140e76fab054aa4aa6ca6ceb9f73fba 2023-04-16 00:56:38,541 DEBUG [StoreCloser-TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/000d29b1df9e4c9f8b4fd8726653f465 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/000d29b1df9e4c9f8b4fd8726653f465 2023-04-16 00:56:38,541 DEBUG [StoreCloser-TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/e5965a0411c44477b420d28bd13dd872 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/e5965a0411c44477b420d28bd13dd872 2023-04-16 00:56:38,542 DEBUG [StoreCloser-TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/f62174af8c2047a3b8ad29743a771078 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/f62174af8c2047a3b8ad29743a771078 2023-04-16 00:56:38,543 DEBUG [StoreCloser-TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/0454d7b6b16e44c79e571242d0b3a390 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/0454d7b6b16e44c79e571242d0b3a390 2023-04-16 00:56:38,544 DEBUG [StoreCloser-TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/c5e34cd11dd64f82a0c80a478d39dc46 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/c5e34cd11dd64f82a0c80a478d39dc46 2023-04-16 00:56:38,545 DEBUG [StoreCloser-TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/98f28f3e808b4fdabf1fa12b5c18e302 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/98f28f3e808b4fdabf1fa12b5c18e302 2023-04-16 00:56:38,545 DEBUG [StoreCloser-TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/aa952594b8ee4cbd8d7653fcbb8da493 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/aa952594b8ee4cbd8d7653fcbb8da493 2023-04-16 00:56:38,546 DEBUG [StoreCloser-TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/10af9a31afdd44b2bb4c51e276db3426 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/10af9a31afdd44b2bb4c51e276db3426 2023-04-16 00:56:38,547 DEBUG [StoreCloser-TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/e917078e0ff64ebea0587c7db31a293a to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/e917078e0ff64ebea0587c7db31a293a 2023-04-16 00:56:38,548 DEBUG [StoreCloser-TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/7ba5a16a30174fc09fe5b6e4ae542936 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/7ba5a16a30174fc09fe5b6e4ae542936 2023-04-16 00:56:38,548 DEBUG [StoreCloser-TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/c54e596b6b774e48b4ac80b98342a4fd to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/c54e596b6b774e48b4ac80b98342a4fd 2023-04-16 00:56:38,549 DEBUG [StoreCloser-TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/c8f6e4654d98421c82dac933bc788866 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/c8f6e4654d98421c82dac933bc788866 2023-04-16 00:56:38,550 DEBUG [StoreCloser-TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/c5d3816f8b4d4d149e50042d81e8ba7b to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/c5d3816f8b4d4d149e50042d81e8ba7b 2023-04-16 00:56:38,551 DEBUG [StoreCloser-TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/e21e02618d174623b613a8d1163f365f to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/e21e02618d174623b613a8d1163f365f 2023-04-16 00:56:38,551 DEBUG [StoreCloser-TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/36121c3b7ff04cf09f73986d1ff097e3 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/36121c3b7ff04cf09f73986d1ff097e3 2023-04-16 00:56:38,552 DEBUG [StoreCloser-TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/1b14d0512fb54b2ca3808faeebf965a6 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/1b14d0512fb54b2ca3808faeebf965a6 2023-04-16 00:56:38,553 DEBUG [StoreCloser-TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/2e98666f01c24e69a099e8eee156ee6e to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/2e98666f01c24e69a099e8eee156ee6e 2023-04-16 00:56:38,554 DEBUG [StoreCloser-TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/c95993956ef44794ab016c69f0c94946 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/c95993956ef44794ab016c69f0c94946 2023-04-16 00:56:38,554 DEBUG [StoreCloser-TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/007c353b19c74958a6b7f5cc4c51fece to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/007c353b19c74958a6b7f5cc4c51fece 2023-04-16 00:56:38,555 DEBUG [StoreCloser-TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/f59a5489aa924fe2a05e495dc0ee4e2c to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/f59a5489aa924fe2a05e495dc0ee4e2c 2023-04-16 00:56:38,556 DEBUG [StoreCloser-TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/703ddd1cbced43b287286c523fcea3fa to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/703ddd1cbced43b287286c523fcea3fa 2023-04-16 00:56:38,557 DEBUG [StoreCloser-TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/a1676505a0984a60834a0f721fd8c557 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/a1676505a0984a60834a0f721fd8c557 2023-04-16 00:56:38,558 DEBUG [StoreCloser-TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/78e3e91f3b5e4ea997d8229031a4ed0c to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/78e3e91f3b5e4ea997d8229031a4ed0c 2023-04-16 00:56:38,559 DEBUG [StoreCloser-TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/5455b9b925c2464aa6425061dea32d4a to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/5455b9b925c2464aa6425061dea32d4a 2023-04-16 00:56:38,560 DEBUG [StoreCloser-TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/a17dd28a2c0243fbba5a7c589e5dce26 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/a17dd28a2c0243fbba5a7c589e5dce26 2023-04-16 00:56:38,561 DEBUG [StoreCloser-TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/4a9ae918db844b738da120a5a9eef156 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/4a9ae918db844b738da120a5a9eef156 2023-04-16 00:56:38,562 DEBUG [StoreCloser-TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/fb0afb1309664563a14a31e108cd473b to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/fb0afb1309664563a14a31e108cd473b 2023-04-16 00:56:38,563 DEBUG [StoreCloser-TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/f8a86b4f975d46ce9c525645e6a5aeb5 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/f8a86b4f975d46ce9c525645e6a5aeb5 2023-04-16 00:56:38,564 DEBUG [StoreCloser-TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/5b5b4eb2e1ba434b9d99bf5de9824275 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/5b5b4eb2e1ba434b9d99bf5de9824275 2023-04-16 00:56:38,567 DEBUG [StoreCloser-TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/124b949b8e2c4623b6d3c107c5e15ac6, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/edc54e20ad384b238b805bdb755bbfe4, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/c4a61b44ad3345e48ed0e12d54c2a06f, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/17562f43b6914cfca9e885928bcf4d9f, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/acbdf8b5ad7640feacb35f81653f8da2, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/c0dac761019a47d4a1a0ab2d45f5f87d, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/ffe29d2d730248dd8a659ca42454aa97, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/853941a538f34158835a61f3098bdaba, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/8d8c35a7c8bf42e4862f34c5e071664b, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/259d737a359841719c910c2f0e8fc933, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/61233cb6892543d7a2bf50475066ea37, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/432c54ce7abc4f13b6ca52483ff9189e, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/0b150b353ab549cd98002fc17a4fb298, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/0ab1f2532b1c4d8bbab20bddceec9d47, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/b3f9b6a3d4514fac89ae7134e1859c8a, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/8546acaf4e1e47f0ba2fb98ced02e994, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/e645f6b6bf834a8fab0c6977593479d9, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/d27faa9987d741c48a2b7d699ff5c889, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/10dcebe0daf24fecb73a2e5e4ea14010, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/03425eb8de8e46f78c8746cc7ef3baeb, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/8ab0b6d870784a6b9aa7ee9be8159c36, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/058046d40cea48548d46e33010220d87, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/26bc504829a9431ba5e28f9fd59417f0, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/fa97bea0af4e4f5491453e35a38bd085, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/2a69af8ce41f4ef0a8a99865999bf042, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/58df185bf1f14e7ebda08f4cc70d58d4, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/6d70239dc59141e4b22c0a82755cf093, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/4417e886af8f4637b7e2ecd8d778bf63, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/597fc8d38ece43eab4d4c09908966f14, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/1617ddb85e8f4717942b9cc8524fba3f, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/f499850510f649b0963b8e24ed9becea, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/6707b49b39784d269b461831ac356ada, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/405666d06d51430c89c17cabe8ad2ac3, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/1ca76d69bd8841bd947f25033ae474d3, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/75e75e6448fb4b3faebf20cf9b1e65e3, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/e4ed3fb509e94ee294acb2f9a36aada0, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/2bbb9b6739aa4649a9850bd0ce9ac919, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/27642fd671a84c598fc9bba20080fbb3, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/0cfe7ca6635a4b31a3eabc5368f5aec1, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/c4081d2ea77c47b4aeeeb18f883af435, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/999304bf7b3e4156b92df1e201a7bffc, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/3a886566faae4278a01b679f361f642b, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/d20d89a64451429d9773f8961222a74e, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/f146ab84266e4546ad715eac04e8bd38, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/e3a71fdf0aa0416a976512e802b0ad27, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/4492b0b9e8c64fef9380ae8a5f8af5ad, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/6b64c1550c544cd099f3c8a1b84b780d, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/ea3f2d4b75654f28aeca92dc38cec363, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/261635fdb2db4c0e89cf7fb6264e69fe] to archive 2023-04-16 00:56:38,568 DEBUG [StoreCloser-TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-16 00:56:38,570 DEBUG [StoreCloser-TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/124b949b8e2c4623b6d3c107c5e15ac6 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/124b949b8e2c4623b6d3c107c5e15ac6 2023-04-16 00:56:38,570 DEBUG [StoreCloser-TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/edc54e20ad384b238b805bdb755bbfe4 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/edc54e20ad384b238b805bdb755bbfe4 2023-04-16 00:56:38,571 DEBUG [StoreCloser-TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/c4a61b44ad3345e48ed0e12d54c2a06f to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/c4a61b44ad3345e48ed0e12d54c2a06f 2023-04-16 00:56:38,572 DEBUG [StoreCloser-TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/17562f43b6914cfca9e885928bcf4d9f to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/17562f43b6914cfca9e885928bcf4d9f 2023-04-16 00:56:38,573 DEBUG [StoreCloser-TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/acbdf8b5ad7640feacb35f81653f8da2 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/acbdf8b5ad7640feacb35f81653f8da2 2023-04-16 00:56:38,574 DEBUG [StoreCloser-TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/c0dac761019a47d4a1a0ab2d45f5f87d to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/c0dac761019a47d4a1a0ab2d45f5f87d 2023-04-16 00:56:38,575 DEBUG [StoreCloser-TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/ffe29d2d730248dd8a659ca42454aa97 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/ffe29d2d730248dd8a659ca42454aa97 2023-04-16 00:56:38,575 DEBUG [StoreCloser-TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/853941a538f34158835a61f3098bdaba to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/853941a538f34158835a61f3098bdaba 2023-04-16 00:56:38,576 DEBUG [StoreCloser-TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/8d8c35a7c8bf42e4862f34c5e071664b to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/8d8c35a7c8bf42e4862f34c5e071664b 2023-04-16 00:56:38,577 DEBUG [StoreCloser-TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/259d737a359841719c910c2f0e8fc933 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/259d737a359841719c910c2f0e8fc933 2023-04-16 00:56:38,578 DEBUG [StoreCloser-TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/61233cb6892543d7a2bf50475066ea37 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/61233cb6892543d7a2bf50475066ea37 2023-04-16 00:56:38,578 DEBUG [StoreCloser-TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/432c54ce7abc4f13b6ca52483ff9189e to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/432c54ce7abc4f13b6ca52483ff9189e 2023-04-16 00:56:38,579 DEBUG [StoreCloser-TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/0b150b353ab549cd98002fc17a4fb298 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/0b150b353ab549cd98002fc17a4fb298 2023-04-16 00:56:38,579 DEBUG [StoreCloser-TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/0ab1f2532b1c4d8bbab20bddceec9d47 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/0ab1f2532b1c4d8bbab20bddceec9d47 2023-04-16 00:56:38,580 DEBUG [StoreCloser-TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/b3f9b6a3d4514fac89ae7134e1859c8a to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/b3f9b6a3d4514fac89ae7134e1859c8a 2023-04-16 00:56:38,581 DEBUG [StoreCloser-TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/8546acaf4e1e47f0ba2fb98ced02e994 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/8546acaf4e1e47f0ba2fb98ced02e994 2023-04-16 00:56:38,581 DEBUG [StoreCloser-TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/e645f6b6bf834a8fab0c6977593479d9 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/e645f6b6bf834a8fab0c6977593479d9 2023-04-16 00:56:38,582 DEBUG [StoreCloser-TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/d27faa9987d741c48a2b7d699ff5c889 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/d27faa9987d741c48a2b7d699ff5c889 2023-04-16 00:56:38,583 DEBUG [StoreCloser-TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/10dcebe0daf24fecb73a2e5e4ea14010 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/10dcebe0daf24fecb73a2e5e4ea14010 2023-04-16 00:56:38,583 DEBUG [StoreCloser-TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/03425eb8de8e46f78c8746cc7ef3baeb to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/03425eb8de8e46f78c8746cc7ef3baeb 2023-04-16 00:56:38,584 DEBUG [StoreCloser-TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/8ab0b6d870784a6b9aa7ee9be8159c36 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/8ab0b6d870784a6b9aa7ee9be8159c36 2023-04-16 00:56:38,584 DEBUG [StoreCloser-TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/058046d40cea48548d46e33010220d87 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/058046d40cea48548d46e33010220d87 2023-04-16 00:56:38,585 DEBUG [StoreCloser-TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/26bc504829a9431ba5e28f9fd59417f0 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/26bc504829a9431ba5e28f9fd59417f0 2023-04-16 00:56:38,586 DEBUG [StoreCloser-TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/fa97bea0af4e4f5491453e35a38bd085 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/fa97bea0af4e4f5491453e35a38bd085 2023-04-16 00:56:38,586 DEBUG [StoreCloser-TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/2a69af8ce41f4ef0a8a99865999bf042 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/2a69af8ce41f4ef0a8a99865999bf042 2023-04-16 00:56:38,587 DEBUG [StoreCloser-TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/58df185bf1f14e7ebda08f4cc70d58d4 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/58df185bf1f14e7ebda08f4cc70d58d4 2023-04-16 00:56:38,587 DEBUG [StoreCloser-TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/6d70239dc59141e4b22c0a82755cf093 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/6d70239dc59141e4b22c0a82755cf093 2023-04-16 00:56:38,588 DEBUG [StoreCloser-TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/4417e886af8f4637b7e2ecd8d778bf63 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/4417e886af8f4637b7e2ecd8d778bf63 2023-04-16 00:56:38,589 DEBUG [StoreCloser-TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/597fc8d38ece43eab4d4c09908966f14 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/597fc8d38ece43eab4d4c09908966f14 2023-04-16 00:56:38,589 DEBUG [StoreCloser-TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/1617ddb85e8f4717942b9cc8524fba3f to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/1617ddb85e8f4717942b9cc8524fba3f 2023-04-16 00:56:38,590 DEBUG [StoreCloser-TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/f499850510f649b0963b8e24ed9becea to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/f499850510f649b0963b8e24ed9becea 2023-04-16 00:56:38,591 DEBUG [StoreCloser-TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/6707b49b39784d269b461831ac356ada to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/6707b49b39784d269b461831ac356ada 2023-04-16 00:56:38,592 DEBUG [StoreCloser-TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/405666d06d51430c89c17cabe8ad2ac3 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/405666d06d51430c89c17cabe8ad2ac3 2023-04-16 00:56:38,592 DEBUG [StoreCloser-TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/1ca76d69bd8841bd947f25033ae474d3 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/1ca76d69bd8841bd947f25033ae474d3 2023-04-16 00:56:38,593 DEBUG [StoreCloser-TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/75e75e6448fb4b3faebf20cf9b1e65e3 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/75e75e6448fb4b3faebf20cf9b1e65e3 2023-04-16 00:56:38,594 DEBUG [StoreCloser-TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/e4ed3fb509e94ee294acb2f9a36aada0 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/e4ed3fb509e94ee294acb2f9a36aada0 2023-04-16 00:56:38,595 DEBUG [StoreCloser-TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/2bbb9b6739aa4649a9850bd0ce9ac919 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/2bbb9b6739aa4649a9850bd0ce9ac919 2023-04-16 00:56:38,595 DEBUG [StoreCloser-TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/27642fd671a84c598fc9bba20080fbb3 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/27642fd671a84c598fc9bba20080fbb3 2023-04-16 00:56:38,596 DEBUG [StoreCloser-TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/0cfe7ca6635a4b31a3eabc5368f5aec1 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/0cfe7ca6635a4b31a3eabc5368f5aec1 2023-04-16 00:56:38,597 DEBUG [StoreCloser-TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/c4081d2ea77c47b4aeeeb18f883af435 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/c4081d2ea77c47b4aeeeb18f883af435 2023-04-16 00:56:38,598 DEBUG [StoreCloser-TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/999304bf7b3e4156b92df1e201a7bffc to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/999304bf7b3e4156b92df1e201a7bffc 2023-04-16 00:56:38,598 DEBUG [StoreCloser-TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/3a886566faae4278a01b679f361f642b to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/3a886566faae4278a01b679f361f642b 2023-04-16 00:56:38,599 DEBUG [StoreCloser-TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/d20d89a64451429d9773f8961222a74e to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/d20d89a64451429d9773f8961222a74e 2023-04-16 00:56:38,600 DEBUG [StoreCloser-TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/f146ab84266e4546ad715eac04e8bd38 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/f146ab84266e4546ad715eac04e8bd38 2023-04-16 00:56:38,601 DEBUG [StoreCloser-TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/e3a71fdf0aa0416a976512e802b0ad27 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/e3a71fdf0aa0416a976512e802b0ad27 2023-04-16 00:56:38,601 DEBUG [StoreCloser-TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/4492b0b9e8c64fef9380ae8a5f8af5ad to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/4492b0b9e8c64fef9380ae8a5f8af5ad 2023-04-16 00:56:38,602 DEBUG [StoreCloser-TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/6b64c1550c544cd099f3c8a1b84b780d to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/6b64c1550c544cd099f3c8a1b84b780d 2023-04-16 00:56:38,603 DEBUG [StoreCloser-TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/ea3f2d4b75654f28aeca92dc38cec363 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/ea3f2d4b75654f28aeca92dc38cec363 2023-04-16 00:56:38,603 DEBUG [StoreCloser-TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/261635fdb2db4c0e89cf7fb6264e69fe to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/261635fdb2db4c0e89cf7fb6264e69fe 2023-04-16 00:56:38,607 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/recovered.edits/1338.seqid, newMaxSeqId=1338, maxSeqId=4 2023-04-16 00:56:38,608 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec. 2023-04-16 00:56:38,608 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for d538367277459b70f4a06235dfaacfec: 2023-04-16 00:56:38,609 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed d538367277459b70f4a06235dfaacfec 2023-04-16 00:56:38,610 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=63 updating hbase:meta row=d538367277459b70f4a06235dfaacfec, regionState=CLOSED 2023-04-16 00:56:38,610 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681606598610"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681606598610"}]},"ts":"1681606598610"} 2023-04-16 00:56:38,611 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=64, resume processing ppid=63 2023-04-16 00:56:38,611 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=64, ppid=63, state=SUCCESS; CloseRegionProcedure d538367277459b70f4a06235dfaacfec, server=jenkins-hbase4.apache.org,34975,1681606419217 in 3.3320 sec 2023-04-16 00:56:38,613 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=63, resume processing ppid=62 2023-04-16 00:56:38,613 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=63, ppid=62, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=d538367277459b70f4a06235dfaacfec, UNASSIGN in 3.3350 sec 2023-04-16 00:56:38,614 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681606598614"}]},"ts":"1681606598614"} 2023-04-16 00:56:38,614 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-16 00:56:38,616 INFO [PEWorker-5] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-16 00:56:38,617 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=62, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 3.4420 sec 2023-04-16 00:56:39,283 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-16 00:56:39,283 INFO [Listener at localhost/33477] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 62 completed 2023-04-16 00:56:39,283 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete TestAcidGuarantees 2023-04-16 00:56:39,284 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] procedure2.ProcedureExecutor(1029): Stored pid=65, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-16 00:56:39,286 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=65, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-16 00:56:39,286 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1227): Checking to see if procedure is done pid=65 2023-04-16 00:56:39,286 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=65, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-16 00:56:39,289 DEBUG [HFileArchiver-13] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/.tmp/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec 2023-04-16 00:56:39,291 DEBUG [HFileArchiver-13] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/.tmp/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A, FileablePath, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/.tmp/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B, FileablePath, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/.tmp/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C, FileablePath, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/.tmp/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/recovered.edits] 2023-04-16 00:56:39,293 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/.tmp/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A/42eab723fbfc4befb3a19c59e08d0032 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A/42eab723fbfc4befb3a19c59e08d0032 2023-04-16 00:56:39,294 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/.tmp/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A/a3070582caca409c81b33e41f756f35c to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A/a3070582caca409c81b33e41f756f35c 2023-04-16 00:56:39,295 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/.tmp/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A/ef9f5ecad71e408bba6d3eda91b99c1f to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/A/ef9f5ecad71e408bba6d3eda91b99c1f 2023-04-16 00:56:39,297 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/.tmp/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/70fe19ce2dc84ba6b40a73c2f0780b84 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/70fe19ce2dc84ba6b40a73c2f0780b84 2023-04-16 00:56:39,298 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/.tmp/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/8985806958fd4bb4a0757e4e45134bb1 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/8985806958fd4bb4a0757e4e45134bb1 2023-04-16 00:56:39,299 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/.tmp/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/b8e276a847b44ebc8c86de137ebffb11 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/B/b8e276a847b44ebc8c86de137ebffb11 2023-04-16 00:56:39,301 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/.tmp/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/2c49b07d3f5a414581ebcc21ffcca00d to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/2c49b07d3f5a414581ebcc21ffcca00d 2023-04-16 00:56:39,302 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/.tmp/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/460c0e2776fc4cc9a516ed393b868a2d to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/460c0e2776fc4cc9a516ed393b868a2d 2023-04-16 00:56:39,302 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/.tmp/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/c54e5cfc0a824d31ad1f163d399b71ac to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/C/c54e5cfc0a824d31ad1f163d399b71ac 2023-04-16 00:56:39,305 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/.tmp/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/recovered.edits/1338.seqid to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec/recovered.edits/1338.seqid 2023-04-16 00:56:39,305 DEBUG [HFileArchiver-13] backup.HFileArchiver(596): Deleted hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/.tmp/data/default/TestAcidGuarantees/d538367277459b70f4a06235dfaacfec 2023-04-16 00:56:39,305 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-16 00:56:39,305 DEBUG [PEWorker-3] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-16 00:56:39,306 DEBUG [PEWorker-3] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A] 2023-04-16 00:56:39,308 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230416057405dfbe9b46dfa9433776618779a1 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230416057405dfbe9b46dfa9433776618779a1 2023-04-16 00:56:39,309 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304160821f03004e74223a4b2cddc235af7d5 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304160821f03004e74223a4b2cddc235af7d5 2023-04-16 00:56:39,310 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304160afcc0ca4dc5474895e4312ebe5c67eb to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304160afcc0ca4dc5474895e4312ebe5c67eb 2023-04-16 00:56:39,311 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304160e39d029a4b54df59cbfbbc7420a3542 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304160e39d029a4b54df59cbfbbc7420a3542 2023-04-16 00:56:39,312 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304161a68509eeba6424a86c07492a45b4c3a to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304161a68509eeba6424a86c07492a45b4c3a 2023-04-16 00:56:39,313 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304161e7d0c225a344fa9945f392c73bf0ca2 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304161e7d0c225a344fa9945f392c73bf0ca2 2023-04-16 00:56:39,314 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041620f8e03ebc734e87ab5d28466379f224 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041620f8e03ebc734e87ab5d28466379f224 2023-04-16 00:56:39,315 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041626c00b2bb38249acba7a88cba3516a19 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041626c00b2bb38249acba7a88cba3516a19 2023-04-16 00:56:39,316 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230416321fe1e1ef574ac38614591a8051cc1f to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230416321fe1e1ef574ac38614591a8051cc1f 2023-04-16 00:56:39,317 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304163a567c6cbeea40d398b6ee1b5c52932a to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304163a567c6cbeea40d398b6ee1b5c52932a 2023-04-16 00:56:39,318 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304163c5b5fdb2a96404ab76a3e861a49bed1 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304163c5b5fdb2a96404ab76a3e861a49bed1 2023-04-16 00:56:39,319 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230416572fffa0922444028877c008cb866454 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230416572fffa0922444028877c008cb866454 2023-04-16 00:56:39,319 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041657ec1724939a48b68580b99520655865 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041657ec1724939a48b68580b99520655865 2023-04-16 00:56:39,320 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230416590e5ea7e5674fcd96e8a4f0eff6b272 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230416590e5ea7e5674fcd96e8a4f0eff6b272 2023-04-16 00:56:39,321 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304166b901ddc4c194501844edcf1d3282c3b to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304166b901ddc4c194501844edcf1d3282c3b 2023-04-16 00:56:39,322 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304166cd68c43b0cc4fd8abd2648397431db4 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304166cd68c43b0cc4fd8abd2648397431db4 2023-04-16 00:56:39,323 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041670cc40e97c0846b8ba66c1aa1a26f06b to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041670cc40e97c0846b8ba66c1aa1a26f06b 2023-04-16 00:56:39,323 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304167af9f126d77f482a8369b7f1ec5f90d5 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304167af9f126d77f482a8369b7f1ec5f90d5 2023-04-16 00:56:39,324 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304167c01a5c4663740dab31544804960d89f to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304167c01a5c4663740dab31544804960d89f 2023-04-16 00:56:39,325 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230416823e82c393b74b8b80b927619c6b0c85 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230416823e82c393b74b8b80b927619c6b0c85 2023-04-16 00:56:39,326 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304168438fabfca784820a64d5ac8cd3053d3 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304168438fabfca784820a64d5ac8cd3053d3 2023-04-16 00:56:39,326 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041686dd15117d8d4fb78a89136c80a1d6a3 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041686dd15117d8d4fb78a89136c80a1d6a3 2023-04-16 00:56:39,327 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304168b83dcd486b74d6f94098f93777e1415 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304168b83dcd486b74d6f94098f93777e1415 2023-04-16 00:56:39,328 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304169255505b063b45588475d1b1ccc8285f to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304169255505b063b45588475d1b1ccc8285f 2023-04-16 00:56:39,329 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041696706b8065ca4ccfaf28b59ff03952b5 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041696706b8065ca4ccfaf28b59ff03952b5 2023-04-16 00:56:39,330 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041697c671b16e084c7fb8d7c4a2b89d1b45 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041697c671b16e084c7fb8d7c4a2b89d1b45 2023-04-16 00:56:39,331 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304169baba9bf33654eaf950b946fa590ce3e to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304169baba9bf33654eaf950b946fa590ce3e 2023-04-16 00:56:39,331 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230416b5f3c80037c8436db353ac66fbabdb16 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230416b5f3c80037c8436db353ac66fbabdb16 2023-04-16 00:56:39,332 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230416ba2c64382fec47b9a178d521ba6617fe to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230416ba2c64382fec47b9a178d521ba6617fe 2023-04-16 00:56:39,333 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230416c10c8b34f3644180ba114a0b6a098a55 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230416c10c8b34f3644180ba114a0b6a098a55 2023-04-16 00:56:39,334 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230416c45656fe152348eebea749bfa79b4141 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230416c45656fe152348eebea749bfa79b4141 2023-04-16 00:56:39,335 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230416d487e7b08a26461c80379a6e28e04b68 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230416d487e7b08a26461c80379a6e28e04b68 2023-04-16 00:56:39,336 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230416ded766dc8fef4a4680a1e57235a514d3 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230416ded766dc8fef4a4680a1e57235a514d3 2023-04-16 00:56:39,337 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230416e1f6be038a26467e9e57a9ca8c1e4652 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230416e1f6be038a26467e9e57a9ca8c1e4652 2023-04-16 00:56:39,338 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230416e6aa741db86d40419795df5682b3ffff to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230416e6aa741db86d40419795df5682b3ffff 2023-04-16 00:56:39,339 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230416e7c7683bacf849e68c5a0cbe7d77f264 to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230416e7c7683bacf849e68c5a0cbe7d77f264 2023-04-16 00:56:39,340 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230416f0fcc683c9d6440985e55077fe7bb69d to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230416f0fcc683c9d6440985e55077fe7bb69d 2023-04-16 00:56:39,341 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230416fcb0cbeb4953451f939d0815734b422f to hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230416fcb0cbeb4953451f939d0815734b422f 2023-04-16 00:56:39,341 DEBUG [PEWorker-3] backup.HFileArchiver(596): Deleted hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-16 00:56:39,343 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=65, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-16 00:56:39,344 WARN [PEWorker-3] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-16 00:56:39,346 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-16 00:56:39,347 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=65, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-16 00:56:39,347 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-16 00:56:39,347 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1681606599347"}]},"ts":"9223372036854775807"} 2023-04-16 00:56:39,348 INFO [PEWorker-3] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-16 00:56:39,348 DEBUG [PEWorker-3] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => d538367277459b70f4a06235dfaacfec, NAME => 'TestAcidGuarantees,,1681606571446.d538367277459b70f4a06235dfaacfec.', STARTKEY => '', ENDKEY => ''}] 2023-04-16 00:56:39,348 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-16 00:56:39,348 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1681606599348"}]},"ts":"9223372036854775807"} 2023-04-16 00:56:39,349 INFO [PEWorker-3] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-16 00:56:39,352 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(130): Finished pid=65, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-16 00:56:39,352 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=65, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 69 msec 2023-04-16 00:56:39,387 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38795] master.MasterRpcServices(1227): Checking to see if procedure is done pid=65 2023-04-16 00:56:39,387 INFO [Listener at localhost/33477] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 65 completed 2023-04-16 00:56:39,398 INFO [Listener at localhost/33477] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testMobGetAtomicity Thread=228 (was 227) - Thread LEAK? -, OpenFileDescriptor=626 (was 622) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=393 (was 419), ProcessCount=171 (was 171), AvailableMemoryMB=1123 (was 1029) - AvailableMemoryMB LEAK? - 2023-04-16 00:56:39,398 INFO [Listener at localhost/33477] hbase.HBaseTestingUtility(1286): Shutting down minicluster 2023-04-16 00:56:39,399 INFO [Listener at localhost/33477] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-16 00:56:39,399 DEBUG [Listener at localhost/33477] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6ed65498 to 127.0.0.1:53150 2023-04-16 00:56:39,399 DEBUG [Listener at localhost/33477] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-16 00:56:39,400 DEBUG [Listener at localhost/33477] util.JVMClusterUtil(237): Shutting down HBase Cluster 2023-04-16 00:56:39,400 DEBUG [Listener at localhost/33477] util.JVMClusterUtil(257): Found active master hash=1752698928, stopped=false 2023-04-16 00:56:39,400 INFO [Listener at localhost/33477] master.ServerManager(901): Cluster shutdown requested of master=jenkins-hbase4.apache.org,38795,1681606417971 2023-04-16 00:56:39,401 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-04-16 00:56:39,401 INFO [Listener at localhost/33477] procedure2.ProcedureExecutor(629): Stopping 2023-04-16 00:56:39,401 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-04-16 00:56:39,401 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-16 00:56:39,401 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-04-16 00:56:39,401 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-04-16 00:56:39,401 DEBUG [Listener at localhost/33477] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x0e946445 to 127.0.0.1:53150 2023-04-16 00:56:39,401 DEBUG [Listener at localhost/33477] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-16 00:56:39,402 INFO [Listener at localhost/33477] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase4.apache.org,34975,1681606419217' ***** 2023-04-16 00:56:39,402 INFO [Listener at localhost/33477] regionserver.HRegionServer(2310): STOPPED: Shutdown requested 2023-04-16 00:56:39,402 INFO [RS:0;jenkins-hbase4:34975] regionserver.HeapMemoryManager(220): Stopping 2023-04-16 00:56:39,402 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-04-16 00:56:39,402 INFO [RS:0;jenkins-hbase4:34975] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager gracefully. 2023-04-16 00:56:39,402 INFO [RS:0;jenkins-hbase4:34975] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager gracefully. 2023-04-16 00:56:39,402 INFO [RS:0;jenkins-hbase4:34975] regionserver.HRegionServer(3304): Received CLOSE for bebb4c20d3b4786f542e659d1c13e314 2023-04-16 00:56:39,403 INFO [RS:0;jenkins-hbase4:34975] regionserver.HRegionServer(1145): stopping server jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:39,403 DEBUG [RS:0;jenkins-hbase4:34975] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x62be487e to 127.0.0.1:53150 2023-04-16 00:56:39,404 DEBUG [RS:0;jenkins-hbase4:34975] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-16 00:56:39,404 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing bebb4c20d3b4786f542e659d1c13e314, disabling compactions & flushes 2023-04-16 00:56:39,404 INFO [RS:0;jenkins-hbase4:34975] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-04-16 00:56:39,404 INFO [RS:0;jenkins-hbase4:34975] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-04-16 00:56:39,404 INFO [RS:0;jenkins-hbase4:34975] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-04-16 00:56:39,404 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1681606421982.bebb4c20d3b4786f542e659d1c13e314. 2023-04-16 00:56:39,404 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1681606421982.bebb4c20d3b4786f542e659d1c13e314. 2023-04-16 00:56:39,404 INFO [RS:0;jenkins-hbase4:34975] regionserver.HRegionServer(3304): Received CLOSE for 1588230740 2023-04-16 00:56:39,404 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1681606421982.bebb4c20d3b4786f542e659d1c13e314. after waiting 0 ms 2023-04-16 00:56:39,404 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1681606421982.bebb4c20d3b4786f542e659d1c13e314. 2023-04-16 00:56:39,404 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing bebb4c20d3b4786f542e659d1c13e314 1/1 column families, dataSize=78 B heapSize=488 B 2023-04-16 00:56:39,404 INFO [RS:0;jenkins-hbase4:34975] regionserver.HRegionServer(1475): Waiting on 2 regions to close 2023-04-16 00:56:39,404 DEBUG [RS:0;jenkins-hbase4:34975] regionserver.HRegionServer(1479): Online Regions={1588230740=hbase:meta,,1.1588230740, bebb4c20d3b4786f542e659d1c13e314=hbase:namespace,,1681606421982.bebb4c20d3b4786f542e659d1c13e314.} 2023-04-16 00:56:39,405 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-04-16 00:56:39,405 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 2023-04-16 00:56:39,405 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-04-16 00:56:39,405 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-04-16 00:56:39,405 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 2023-04-16 00:56:39,405 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 1588230740 3/3 column families, dataSize=20.92 KB heapSize=36.24 KB 2023-04-16 00:56:39,406 DEBUG [RS:0;jenkins-hbase4:34975] regionserver.HRegionServer(1505): Waiting on 1588230740, bebb4c20d3b4786f542e659d1c13e314 2023-04-16 00:56:39,433 INFO [regionserver/jenkins-hbase4:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-04-16 00:56:39,534 INFO [regionserver/jenkins-hbase4:0.Chore.1] hbase.ScheduledChore(146): Chore: MemstoreFlusherChore was stopped 2023-04-16 00:56:39,534 INFO [regionserver/jenkins-hbase4:0.Chore.1] hbase.ScheduledChore(146): Chore: CompactionChecker was stopped 2023-04-16 00:56:39,606 DEBUG [RS:0;jenkins-hbase4:34975] regionserver.HRegionServer(1505): Waiting on 1588230740, bebb4c20d3b4786f542e659d1c13e314 2023-04-16 00:56:39,807 DEBUG [RS:0;jenkins-hbase4:34975] regionserver.HRegionServer(1505): Waiting on 1588230740, bebb4c20d3b4786f542e659d1c13e314 2023-04-16 00:56:39,816 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=78 B at sequenceid=6 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/hbase/namespace/bebb4c20d3b4786f542e659d1c13e314/.tmp/info/16a3efdadfac42a7898e1ec7fa45a3e3 2023-04-16 00:56:39,816 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=18.26 KB at sequenceid=93 (bloomFilter=false), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/hbase/meta/1588230740/.tmp/info/e96267d8fff949e0a35dc8e9f6fd4081 2023-04-16 00:56:39,821 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/hbase/namespace/bebb4c20d3b4786f542e659d1c13e314/.tmp/info/16a3efdadfac42a7898e1ec7fa45a3e3 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/hbase/namespace/bebb4c20d3b4786f542e659d1c13e314/info/16a3efdadfac42a7898e1ec7fa45a3e3 2023-04-16 00:56:39,821 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for e96267d8fff949e0a35dc8e9f6fd4081 2023-04-16 00:56:39,824 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/hbase/namespace/bebb4c20d3b4786f542e659d1c13e314/info/16a3efdadfac42a7898e1ec7fa45a3e3, entries=2, sequenceid=6, filesize=4.8 K 2023-04-16 00:56:39,825 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~78 B/78, heapSize ~472 B/472, currentSize=0 B/0 for bebb4c20d3b4786f542e659d1c13e314 in 421ms, sequenceid=6, compaction requested=false 2023-04-16 00:56:39,830 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/hbase/namespace/bebb4c20d3b4786f542e659d1c13e314/recovered.edits/9.seqid, newMaxSeqId=9, maxSeqId=1 2023-04-16 00:56:39,831 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed hbase:namespace,,1681606421982.bebb4c20d3b4786f542e659d1c13e314. 2023-04-16 00:56:39,831 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for bebb4c20d3b4786f542e659d1c13e314: 2023-04-16 00:56:39,831 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed hbase:namespace,,1681606421982.bebb4c20d3b4786f542e659d1c13e314. 2023-04-16 00:56:40,007 DEBUG [RS:0;jenkins-hbase4:34975] regionserver.HRegionServer(1505): Waiting on 1588230740 2023-04-16 00:56:40,207 DEBUG [RS:0;jenkins-hbase4:34975] regionserver.HRegionServer(1505): Waiting on 1588230740 2023-04-16 00:56:40,227 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=588 B at sequenceid=93 (bloomFilter=false), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/hbase/meta/1588230740/.tmp/rep_barrier/3cd881bb9b70418383e2784b744df259 2023-04-16 00:56:40,230 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 3cd881bb9b70418383e2784b744df259 2023-04-16 00:56:40,407 INFO [RS:0;jenkins-hbase4:34975] regionserver.HRegionServer(1475): Waiting on 1 regions to close 2023-04-16 00:56:40,408 DEBUG [RS:0;jenkins-hbase4:34975] regionserver.HRegionServer(1479): Online Regions={1588230740=hbase:meta,,1.1588230740} 2023-04-16 00:56:40,408 DEBUG [RS:0;jenkins-hbase4:34975] regionserver.HRegionServer(1505): Waiting on 1588230740 2023-04-16 00:56:40,608 DEBUG [RS:0;jenkins-hbase4:34975] regionserver.HRegionServer(1505): Waiting on 1588230740 2023-04-16 00:56:40,616 DEBUG [regionserver/jenkins-hbase4:0.Chore.1] throttle.PressureAwareCompactionThroughputController(103): CompactionPressure is 0.0, tune throughput to 50.00 MB/second 2023-04-16 00:56:40,636 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=2.08 KB at sequenceid=93 (bloomFilter=false), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/hbase/meta/1588230740/.tmp/table/685b097511164de99f51102a3fb769ad 2023-04-16 00:56:40,639 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 685b097511164de99f51102a3fb769ad 2023-04-16 00:56:40,639 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/hbase/meta/1588230740/.tmp/info/e96267d8fff949e0a35dc8e9f6fd4081 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/hbase/meta/1588230740/info/e96267d8fff949e0a35dc8e9f6fd4081 2023-04-16 00:56:40,642 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for e96267d8fff949e0a35dc8e9f6fd4081 2023-04-16 00:56:40,642 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/hbase/meta/1588230740/info/e96267d8fff949e0a35dc8e9f6fd4081, entries=22, sequenceid=93, filesize=7.1 K 2023-04-16 00:56:40,644 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/hbase/meta/1588230740/.tmp/rep_barrier/3cd881bb9b70418383e2784b744df259 as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/hbase/meta/1588230740/rep_barrier/3cd881bb9b70418383e2784b744df259 2023-04-16 00:56:40,646 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 3cd881bb9b70418383e2784b744df259 2023-04-16 00:56:40,646 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/hbase/meta/1588230740/rep_barrier/3cd881bb9b70418383e2784b744df259, entries=6, sequenceid=93, filesize=5.5 K 2023-04-16 00:56:40,647 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/hbase/meta/1588230740/.tmp/table/685b097511164de99f51102a3fb769ad as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/hbase/meta/1588230740/table/685b097511164de99f51102a3fb769ad 2023-04-16 00:56:40,649 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 685b097511164de99f51102a3fb769ad 2023-04-16 00:56:40,649 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/hbase/meta/1588230740/table/685b097511164de99f51102a3fb769ad, entries=9, sequenceid=93, filesize=5.5 K 2023-04-16 00:56:40,650 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~20.92 KB/21417, heapSize ~36.20 KB/37064, currentSize=0 B/0 for 1588230740 in 1245ms, sequenceid=93, compaction requested=false 2023-04-16 00:56:40,657 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/data/hbase/meta/1588230740/recovered.edits/96.seqid, newMaxSeqId=96, maxSeqId=1 2023-04-16 00:56:40,658 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-04-16 00:56:40,659 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed hbase:meta,,1.1588230740 2023-04-16 00:56:40,659 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-04-16 00:56:40,659 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed hbase:meta,,1.1588230740 2023-04-16 00:56:40,808 INFO [RS:0;jenkins-hbase4:34975] regionserver.HRegionServer(1171): stopping server jenkins-hbase4.apache.org,34975,1681606419217; all regions closed. 2023-04-16 00:56:40,815 DEBUG [RS:0;jenkins-hbase4:34975] wal.AbstractFSWAL(1028): Moved 1 WAL file(s) to /user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/oldWALs 2023-04-16 00:56:40,816 INFO [RS:0;jenkins-hbase4:34975] wal.AbstractFSWAL(1031): Closed WAL: AsyncFSWAL jenkins-hbase4.apache.org%2C34975%2C1681606419217.meta:.meta(num 1681606421645) 2023-04-16 00:56:40,819 DEBUG [RS:0;jenkins-hbase4:34975] wal.AbstractFSWAL(1028): Moved 1 WAL file(s) to /user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/oldWALs 2023-04-16 00:56:40,819 INFO [RS:0;jenkins-hbase4:34975] wal.AbstractFSWAL(1031): Closed WAL: AsyncFSWAL jenkins-hbase4.apache.org%2C34975%2C1681606419217:(num 1681606420674) 2023-04-16 00:56:40,819 DEBUG [RS:0;jenkins-hbase4:34975] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-16 00:56:40,819 INFO [RS:0;jenkins-hbase4:34975] regionserver.LeaseManager(133): Closed leases 2023-04-16 00:56:40,820 INFO [RS:0;jenkins-hbase4:34975] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase4:0 had [ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS, ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS] on shutdown 2023-04-16 00:56:40,820 INFO [regionserver/jenkins-hbase4:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-04-16 00:56:40,821 INFO [RS:0;jenkins-hbase4:34975] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:34975 2023-04-16 00:56:40,826 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-04-16 00:56:40,826 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,34975,1681606419217 2023-04-16 00:56:40,826 ERROR [Listener at localhost/33477-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@26d41885 rejected from java.util.concurrent.ThreadPoolExecutor@22eb8887[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 2803] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-16 00:56:40,827 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-04-16 00:56:40,827 ERROR [Listener at localhost/33477-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@505259ef rejected from java.util.concurrent.ThreadPoolExecutor@22eb8887[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 2803] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-16 00:56:40,828 INFO [RegionServerTracker-0] master.RegionServerTracker(179): RegionServer ephemeral node deleted, processing expiration [jenkins-hbase4.apache.org,34975,1681606419217] 2023-04-16 00:56:40,829 DEBUG [RegionServerTracker-0] master.DeadServer(103): Processing jenkins-hbase4.apache.org,34975,1681606419217; numProcessing=1 2023-04-16 00:56:40,830 DEBUG [RegionServerTracker-0] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase4.apache.org,34975,1681606419217 already deleted, retry=false 2023-04-16 00:56:40,830 INFO [RegionServerTracker-0] master.ServerManager(561): Cluster shutdown set; jenkins-hbase4.apache.org,34975,1681606419217 expired; onlineServers=0 2023-04-16 00:56:40,830 INFO [RegionServerTracker-0] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase4.apache.org,38795,1681606417971' ***** 2023-04-16 00:56:40,830 INFO [RegionServerTracker-0] regionserver.HRegionServer(2310): STOPPED: Cluster shutdown set; onlineServer=0 2023-04-16 00:56:40,830 DEBUG [M:0;jenkins-hbase4:38795] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@29451a0b, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-04-16 00:56:40,830 INFO [M:0;jenkins-hbase4:38795] regionserver.HRegionServer(1145): stopping server jenkins-hbase4.apache.org,38795,1681606417971 2023-04-16 00:56:40,830 INFO [M:0;jenkins-hbase4:38795] regionserver.HRegionServer(1171): stopping server jenkins-hbase4.apache.org,38795,1681606417971; all regions closed. 2023-04-16 00:56:40,830 DEBUG [M:0;jenkins-hbase4:38795] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-16 00:56:40,831 DEBUG [M:0;jenkins-hbase4:38795] cleaner.LogCleaner(198): Cancelling LogCleaner 2023-04-16 00:56:40,831 WARN [OldWALsCleaner-0] cleaner.LogCleaner(186): Interrupted while cleaning old WALs, will try to clean it next round. Exiting. 2023-04-16 00:56:40,831 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1681606420379] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1681606420379,5,FailOnTimeoutGroup] 2023-04-16 00:56:40,831 DEBUG [M:0;jenkins-hbase4:38795] cleaner.HFileCleaner(317): Stopping file delete threads 2023-04-16 00:56:40,831 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1681606420376] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1681606420376,5,FailOnTimeoutGroup] 2023-04-16 00:56:40,831 INFO [M:0;jenkins-hbase4:38795] master.MasterMobCompactionThread(168): Waiting for Mob Compaction Thread to finish... 2023-04-16 00:56:40,831 INFO [M:0;jenkins-hbase4:38795] master.MasterMobCompactionThread(168): Waiting for Region Server Mob Compaction Thread to finish... 2023-04-16 00:56:40,831 INFO [M:0;jenkins-hbase4:38795] hbase.ChoreService(369): Chore service for: master/jenkins-hbase4:0 had [] on shutdown 2023-04-16 00:56:40,832 DEBUG [M:0;jenkins-hbase4:38795] master.HMaster(1512): Stopping service threads 2023-04-16 00:56:40,832 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2023-04-16 00:56:40,832 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-16 00:56:40,832 INFO [M:0;jenkins-hbase4:38795] procedure2.RemoteProcedureDispatcher(118): Stopping procedure remote dispatcher 2023-04-16 00:56:40,832 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-04-16 00:56:40,832 ERROR [M:0;jenkins-hbase4:38795] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] Thread[HFileArchiver-11,5,PEWorkerGroup] Thread[HFileArchiver-12,5,PEWorkerGroup] Thread[HFileArchiver-13,5,PEWorkerGroup] 2023-04-16 00:56:40,833 INFO [M:0;jenkins-hbase4:38795] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-04-16 00:56:40,833 DEBUG [normalizer-worker-0] normalizer.RegionNormalizerWorker(174): interrupt detected. terminating. 2023-04-16 00:56:40,833 DEBUG [M:0;jenkins-hbase4:38795] zookeeper.ZKUtil(398): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Unable to get data of znode /hbase/master because node does not exist (not an error) 2023-04-16 00:56:40,833 WARN [M:0;jenkins-hbase4:38795] master.ActiveMasterManager(326): Failed get of master address: java.io.IOException: Can't get master address from ZooKeeper; znode data == null 2023-04-16 00:56:40,833 INFO [M:0;jenkins-hbase4:38795] assignment.AssignmentManager(315): Stopping assignment manager 2023-04-16 00:56:40,833 INFO [M:0;jenkins-hbase4:38795] region.MasterRegion(167): Closing local region {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, isAbort=false 2023-04-16 00:56:40,834 DEBUG [M:0;jenkins-hbase4:38795] regionserver.HRegion(1604): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-04-16 00:56:40,834 INFO [M:0;jenkins-hbase4:38795] regionserver.HRegion(1626): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-16 00:56:40,834 DEBUG [M:0;jenkins-hbase4:38795] regionserver.HRegion(1647): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-16 00:56:40,834 DEBUG [M:0;jenkins-hbase4:38795] regionserver.HRegion(1714): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-04-16 00:56:40,834 DEBUG [M:0;jenkins-hbase4:38795] regionserver.HRegion(1724): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-16 00:56:40,834 INFO [M:0;jenkins-hbase4:38795] regionserver.HRegion(2745): Flushing 1595e783b53d99cd5eef43b6debb2682 1/1 column families, dataSize=344.65 KB heapSize=392.90 KB 2023-04-16 00:56:40,845 INFO [M:0;jenkins-hbase4:38795] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=344.65 KB at sequenceid=571 (bloomFilter=true), to=hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/e824abf41dd3454e842ceb4f74ca41cf 2023-04-16 00:56:40,849 INFO [M:0;jenkins-hbase4:38795] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for e824abf41dd3454e842ceb4f74ca41cf 2023-04-16 00:56:40,849 DEBUG [M:0;jenkins-hbase4:38795] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/e824abf41dd3454e842ceb4f74ca41cf as hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/e824abf41dd3454e842ceb4f74ca41cf 2023-04-16 00:56:40,852 INFO [M:0;jenkins-hbase4:38795] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for e824abf41dd3454e842ceb4f74ca41cf 2023-04-16 00:56:40,852 INFO [M:0;jenkins-hbase4:38795] regionserver.HStore(1080): Added hdfs://localhost:43665/user/jenkins/test-data/fa60bee3-ffa4-7d9a-b122-1e59ec0886e6/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/e824abf41dd3454e842ceb4f74ca41cf, entries=65, sequenceid=571, filesize=24.7 K 2023-04-16 00:56:40,853 INFO [M:0;jenkins-hbase4:38795] regionserver.HRegion(2948): Finished flush of dataSize ~344.65 KB/352923, heapSize ~392.88 KB/402312, currentSize=0 B/0 for 1595e783b53d99cd5eef43b6debb2682 in 19ms, sequenceid=571, compaction requested=false 2023-04-16 00:56:40,854 INFO [M:0;jenkins-hbase4:38795] regionserver.HRegion(1838): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-16 00:56:40,854 DEBUG [M:0;jenkins-hbase4:38795] regionserver.HRegion(1558): Region close journal for 1595e783b53d99cd5eef43b6debb2682: 2023-04-16 00:56:40,856 INFO [master:store-WAL-Roller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-04-16 00:56:40,856 INFO [M:0;jenkins-hbase4:38795] flush.MasterFlushTableProcedureManager(83): stop: server shutting down. 2023-04-16 00:56:40,857 INFO [M:0;jenkins-hbase4:38795] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:38795 2023-04-16 00:56:40,859 DEBUG [M:0;jenkins-hbase4:38795] zookeeper.RecoverableZooKeeper(172): Node /hbase/rs/jenkins-hbase4.apache.org,38795,1681606417971 already deleted, retry=false 2023-04-16 00:56:40,929 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-04-16 00:56:40,929 INFO [RS:0;jenkins-hbase4:34975] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase4.apache.org,34975,1681606419217; zookeeper connection closed. 2023-04-16 00:56:40,929 ERROR [Listener at localhost/33477-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@281566af rejected from java.util.concurrent.ThreadPoolExecutor@22eb8887[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 2803] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-16 00:56:40,930 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@1074e8f8] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@1074e8f8 2023-04-16 00:56:40,930 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): regionserver:34975-0x1018a7584b80001, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-04-16 00:56:40,930 ERROR [Listener at localhost/33477-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@459951c7 rejected from java.util.concurrent.ThreadPoolExecutor@22eb8887[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 2803] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-16 00:56:40,931 INFO [Listener at localhost/33477] util.JVMClusterUtil(335): Shutdown of 1 master(s) and 1 regionserver(s) complete 2023-04-16 00:56:41,029 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-04-16 00:56:41,029 INFO [M:0;jenkins-hbase4:38795] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase4.apache.org,38795,1681606417971; zookeeper connection closed. 2023-04-16 00:56:41,030 ERROR [Listener at localhost/33477-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@109b9b2 rejected from java.util.concurrent.ThreadPoolExecutor@7c6d3517[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 4683] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-16 00:56:41,030 DEBUG [Listener at localhost/33477-EventThread] zookeeper.ZKWatcher(600): master:38795-0x1018a7584b80000, quorum=127.0.0.1:53150, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-04-16 00:56:41,030 ERROR [Listener at localhost/33477-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@3e31f45a rejected from java.util.concurrent.ThreadPoolExecutor@7c6d3517[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 4683] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-16 00:56:41,031 WARN [Listener at localhost/33477] datanode.DirectoryScanner(534): DirectoryScanner: shutdown has been called 2023-04-16 00:56:41,035 INFO [Listener at localhost/33477] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-04-16 00:56:41,143 WARN [BP-1873989735-172.31.14.131-1681606414745 heartbeating to localhost/127.0.0.1:43665] datanode.IncrementalBlockReportManager(160): IncrementalBlockReportManager interrupted 2023-04-16 00:56:41,143 WARN [BP-1873989735-172.31.14.131-1681606414745 heartbeating to localhost/127.0.0.1:43665] datanode.BPServiceActor(857): Ending block pool service for: Block pool BP-1873989735-172.31.14.131-1681606414745 (Datanode Uuid 1215da9f-09ed-4713-ad5c-9bdf4f6f2014) service to localhost/127.0.0.1:43665 2023-04-16 00:56:41,146 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/eff38a88-f310-5857-a834-b48ec32a7488/cluster_c35e0b5b-fd77-8a80-76dd-b83041c30bf3/dfs/data/data1/current/BP-1873989735-172.31.14.131-1681606414745] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-04-16 00:56:41,146 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/eff38a88-f310-5857-a834-b48ec32a7488/cluster_c35e0b5b-fd77-8a80-76dd-b83041c30bf3/dfs/data/data2/current/BP-1873989735-172.31.14.131-1681606414745] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-04-16 00:56:41,182 INFO [Listener at localhost/33477] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-04-16 00:56:41,356 INFO [Listener at localhost/33477] zookeeper.MiniZooKeeperCluster(344): Shutdown MiniZK cluster with all ZK servers 2023-04-16 00:56:41,544 INFO [Listener at localhost/33477] hbase.HBaseTestingUtility(1293): Minicluster is down